Join 14k builders to the Real-World ML Newsletter ⬇️⬇️⬇️
Text embeddings are vector representations of raw text that you compute using an embedding model
These vectors representations are then used for downstream tasks, like
Classification → for example, to classify tweet sentiment as either positive or negative.
Clustering → for example, to automatically group news into topics.
Retrieval → for example, to find similar documents to a given query.
Retrieval (the “R” in RAG) is the task of finding the most relevant documents given an input query. This is one of the most popular usages of embeddings these days, and the one we focus on in this repository.
There are many embedding models, both open and proprietary, so the question is:
What embedding model is best for you problem? 🤔
In this repository you can find an evaluation script that helps you find the right embedding model for your use case.
To evaluate a model for retrieval using a particular dataset we will
Load the model and your dataset from HuggingFace, with
questions
contexts
, andcorrect answers
Embed the context into the Vector DB, in our case Qdrant.
For each question retrieve the top K
relevant documents from the Vector DB, and
Compare the overlap in information between the retrieved documents and the correct answers
. We will use ragas, an open-source framework for RAG evalution, to compute context precision
and context recall
Finally log the results, so you know what worked best.
You will need
An OpenAI API key, because ragas will be making calls to GPT-3.5 Turbo
to evaluate the context precision and recall.
A Qdrant Vector DB with its corresponding URL and API key, which you can get for FREE by signing up here
Create an .env
file and paste your OPENAI_API_KEY
, QDRANT_URL
and QDRANT_API_KEY
$ cp .env.example .env
Create the virtual environment with Python Poetry
$ make install
Update the list of models and datasets you want to test in config.yml
Run the evaluations
$ make run-evals
Join 15k builders to the Real-World ML Newsletter.
Every Saturday morning.
For FREE.