Table of Contents
Retrieval-Augmented Generation (RAG) represents a cutting-edge innovation in natural language processing (NLP), merging the prowess of retrieval models with generative models to produce context-rich, coherent text. Unlike conventional models that rely solely on pre-trained knowledge, RAG actively retrieves external, up-to-date information to enhance the generation of responses, making it exceptionally suited for applications requiring dynamic and accurate content generation.
RAG operates in a two-phase approach: the retrieval phase and the content generation phase. Initially, algorithms search for information relevant to the user's query from external sources. The generative language model, such as GPT, then uses this context to produce responses, ensuring accuracy by conditioning the generated content on the retrieved data.
In order to facilitate the retrieval portion of RAG, we must choose a vector database in order to conduct a similarity search using the vector generated from the user input, and the precomputed vectors for each of our pieces of external information. That way, we can decide which are most relevant to provide to the generation step. Some considerations in choosing the correct database:
<aside> 📈 Performance and Scalability The database should offer high performance in terms of query speed and accuracy. It must also scale easily to accommodate growing data volumes without a significant drop in performance.
</aside>
<aside> <img src="/icons/filtered_blue.svg" alt="/icons/filtered_blue.svg" width="40px" /> Metadata Filtering Most modern vector databases allow the ability to filter on particular metadata associated with the documents you ingest. However, not all systems allow results to be scored dynamically based on anything other than vector similarity. It’s important to note during your design process if you’ll need capabilities to introduce custom result scoring formulas.
</aside>
<aside> 🤝🏼 Integration Capabilities The ease with which the database can be integrated into your existing systems is important. Look for databases that offer robust APIs, SDKs, and support for the programming languages or frameworks you use. In particular, we recommend choosing databases with LangChain or LlamaIndex integrations, as those are the two leading frameworks for LLM application development.
</aside>
<aside> 💵 Cost Consider both the initial setup cost and the ongoing operational costs. Some vector databases offer a free tier, which can be suitable for development or small-scale applications, while scaling up may incur costs.
</aside>
<aside> 🔒 Security and Compliance Ensure the database meets your security requirements and complies with relevant data protection regulations, especially if you are handling sensitive or personal data.
</aside>
At the time of writing this article, PressW recommends considering Qdrant and Xata.
<aside> <img src="https://prod-files-secure.s3.us-west-2.amazonaws.com/19bd7778-897b-49c0-bb6d-fdbe2db8210c/a932f5a6-22e3-404d-924d-af908897d468/qdrant.svg" alt="https://prod-files-secure.s3.us-west-2.amazonaws.com/19bd7778-897b-49c0-bb6d-fdbe2db8210c/a932f5a6-22e3-404d-924d-af908897d468/qdrant.svg" width="40px" /> Qdrant https://qdrant.tech/
We choose Qdrant when we need to automatically track document lineage and update documents in our database via LangChain Record Manager.
</aside>
<aside> <img src="https://prod-files-secure.s3.us-west-2.amazonaws.com/19bd7778-897b-49c0-bb6d-fdbe2db8210c/7b9a9421-c761-4944-8449-a042d8c26c94/xata.svg" alt="https://prod-files-secure.s3.us-west-2.amazonaws.com/19bd7778-897b-49c0-bb6d-fdbe2db8210c/7b9a9421-c761-4944-8449-a042d8c26c94/xata.svg" width="40px" /> Xata https://xata.io/
We choose Xata when our data outside of the document vectors is highly relational, as it’s a clever combination of PostgreSQL and Vector DB architectures.
</aside>
Effective RAG implementation hinges on fine-tuning your system for contextual understanding and achieving a balance between the retrieved information and the generative model's creativity. This involves ensuring the data quality and relevance, as well as considering ethical implications and bias mitigation in the data used.
RAG Pipelines have a tremendous amount of potential customization to your use case. Those can generally be broken down into a few major categories:
Creating an effective RAG pipeline doesn’t rely solely on optimizing retrieval. It’s best to go back and forth between improving retrieval, understanding the implications of those changes on your system, and tuning your original LLM prompt(s) accordingly. An simple example:
That’s not all though… this could be an entire article in and of itself (hint: coming soon), but for now here’s another sampling of prompting techniques to use with RAG.
This is a strategy that’s usually low effort, but provides a lot of lift. Essentially, you use a model to reorder a subset of your larger document set. However, this is done using the input query and the raw document as the inputs to the process that determines the similarity score, rather than a precomputed embedding of each. This gives the benefit of considering the query in context of the actual document, rather than leveraging a precomputed meaning that doesn’t necessarily apply.
Do note, this is much slower than traditional vector search, which is why we only choose a subset of the most relevant documents as determined by an initial vector search over the precomputed embeddings.
Cohere offers a powerful reranking API that is commonly used for this task:
The embedding model you use is one of the most important steps in RAG! Essentially, it’s what determines what meanings get encoded into each vector when transforming from input text to the fixed length this of numbers that these systems use to determine text similarity.
There are two main things you can do here:
Select a Case Specific Model
There are many pretrained embedding models that have already been tuned on specific terminology, and understand a different specialization of words (think medical, legal, etc.), or even have been tuned directly on a task (although it’s unlikely the task is the exact one you’re looking to accomplish)!
Fine Tune your Own Embedding Model
This option is more complicated, but sometimes necessary for extremely complex retrieval cases, or cases where your actual object needs to be encoded into the retriever. You can pick your own dataset of terms, and allow the model to index the specific terms you’ll need when it sees them utilized in context.
As mentioned above, these models can also be tuned directly on the task. Essentially, you use the embeddings downstream on your task, then you measure their performance on the task using some objective function. The actual performance is then weighed against the optimal performance, and that error signal is propagated back up the chain to the embedding model itself, which updates its weights. The result should be a model that is better at generating embeddings that retrieve the most relevant documents for your task.
In order to prepare documents for intake into the pipeline, you’ll need to chunk them up into manageable pieces. The exact settings for this are relatively task dependent, but here we discuss how to pick a good starting point:
Document Chunking - Choosing Settings
Beyond that, there are a few more common parameters to consider. “Top-K” is on you’ll likely contend with, and that’s just tuning the count of documents returned with each query. You can also configure your system to return a variable amount of documents, as long as their similarity score is above a certain threshold. We’ll dive deeper on this in our upcoming advanced RAG article!
The process of evaluating RAG results can be broken down into a few core actions:
Before pen hits the paper on code, it is important that you understand the success criteria of your system. Build a dataset alongside a subject matter expert (SME) of queries and a set of documents. Then, map the queries to the documents the queries are expected to return.
Now, you are able to use automated evaluation metrics to check how we’ll you’re doing at retrieval every time you make a change to the system. These metrics can measure things such as:
You can even set up an additional LLM pipeline to subjectively rate the relevance of your returned documents to the task at hand leveraging an “executor/observer” framework. Essentially, you can show the second LLM your input prompt, and ask it how relevant it thinks the documents are to the input query and the task we’re trying to execute!
This is your first line and last of defense. The developer, along with a should review the documents and document chunks returned by the RAG system in certain circumstances. When you make changes, the automated scoring will always run. However, in the case of low scores on certain queries, in most cases you can determine what the cause of the low retrieval performance is upon manual verification of the returned results.
Today, there are a few frameworks around to assess the quality of LLM outputs in general, but they also have some specific utilities associated with RAG. Check out RAGAS and LangSmith to jumpstart your evaluation.