Introduction to Retrieval Augmented Generators (RAG): Enhancing Virtual Assistants with Domain-Specific Knowledge

Sascha Gstir
Sascha Gstir · Co-Founder, Pondhouse Data
Published 5 min read
Introduction to Retrieval Augmented Generators (RAG): Enhancing Virtual Assistants with Domain-Specific Knowledge
On this page

Retrieval Augmented Generation (RAG) gives a virtual assistant built on a large language model (LLM) access to proprietary, private and domain-specific knowledge without training that knowledge into the model. The knowledge lives in a vector database. For each question, the system retrieves the most relevant pieces and hands them to the LLM together with the question, and the LLM answers from that material.

This post builds on our earlier article, "Integrating enterprise knowledge with LLMs". We still think RAG is the most effective and precise way to make existing knowledge easy to access, which is why it sits at the core of PondhouseAI. New technologies meet skepticism until they prove useful, so below we go through the capabilities a RAG system needs and its base architecture, then explain why we prefer it over training a model.

What a RAG system needs to do

The traditional way to teach a model specific knowledge is to train or fine-tune it: all relevant data goes into a single model, which retains it in its internal weights. This can perform well, but it brings operational and financial problems. Data preparation is complex, training takes a long time and gets expensive, and it is hard to see why the model knows what it knows. Professionals also have to train and test the model extensively to make sure it improves and still behaves as expected.

RAG drops the idea of putting all knowledge into one large model, because for most companies and applications the drawbacks are too big. Instead, it splits the job into a process with separate architectural components.

Take the user's point of view. Say you are a technical support engineer at a company that builds machinery. You want a virtual assistant that gives fast, precise answers to the issues customers report, and you want to talk to it in natural language. When you ask "When does alarm 5334 occur for machine type XYZ?", the assistant has to understand the question, recall the relevant information it was given earlier and write a suitable answer.

The four capabilities of a RAG system and how they connect

Seen from a high level, that takes four capabilities:

  • Learning means taking in new information, processing it and integrating it with the knowledge already in place.
  • Memorizing means keeping that information accessible so it can be "remembered" when needed. You could argue memorizing is part of learning, but it plays such a central role in RAG that we list it separately.
  • Associating means working out which parts of the available knowledge are relevant to the current question or task.
  • Communicating means processing the question and presenting the needed information in a way that answers it, so people can work with the system easily.

The RAG base architecture, step by step

RAG architecture with ingestion steps A-D and query steps 1-5

Here is how a RAG system gets these four capabilities in technical terms. The diagram labels the ingestion path with letters (A-D) and the query path with numbers (1-5).

"Learning" is the data ingestion and preparation pipeline (A-D). It processes every file that holds knowledge the system should absorb: PDFs, Word documents, CSVs, Markdown files, exports from ticketing systems, database connections and more (A). The raw files go through an information extraction step (B), which uses several techniques to pull out the content that matters (our post on LLM document extraction covers one of them). The extracted text is then split into much smaller units called chunks (C). Each chunk holds a piece of information the system can retrieve later when it is relevant to a query. How you chunk affects retrieval quality; late chunking is one way to improve it.

Judging relevance from a chunk's raw text is hard, so RAG converts each chunk into a representation of its meaning. An embedding model turns the text into a numerical vector called an embedding. Texts with similar meanings get similar embeddings, and texts with different meanings get embeddings that differ a lot.

The embeddings are stored together with the original text in a vector database (D). A vector database is built to store and run operations on these large vectors. It acts as the system's memory.

When a user sends a request (1), the same embedding model turns it into an embedding. The system uses that embedding to search the vector database for chunks with similar embeddings (2). For "When does alarm 5334 occur for machine type XYZ?", it looks for information about "machine type XYZ" and "alarm 5334".

Why use embedding models and a vector database instead of searching for the exact terms "machine type XYZ" and "alarm 5334"? Exact-term search only finds text phrased the same way as the query, so results are often poor. Semantic search over embeddings works on meaning, so the exact wording matters much less and it copes well with different phrasings. Semantic search has a weak spot, though: very specific terms and character sequences like alarm codes, because it looks for similarity rather than exact matches. To cover those, RAG systems sometimes use hybrid search, which combines semantic search with keyword matching. Our Azure AI Search RAG tutorial shows a managed service that supports both.

The search returns the relevant chunks (3). The system combines their original text with the user's question and sends both to the LLM (4). The LLM uses the retrieved text as reference material for its answer, which is where the "retrieval augmented" part of the name comes from.

In the final step, the LLM does what it is good at: it understands the context of the question, picks the information it needs from the reference material and writes the response (5).

Benefits of RAG over training knowledge into the model

The RAG architecture looks complex at first, but each component has one specific, limited job, and that is where its advantages come from. Each component can be improved on its own. Because the knowledge sits in a database rather than inside the LLM, updating it is simple and takes effect immediately. There is no expensive, slow cycle of training and testing the LLM, so you can use pre-trained models as they are without changing their proven, validated behavior. And every stage is fully transparent, which a monolithic LLM can't offer.

Once a RAG pipeline is running, the next step is measuring how well it answers. Our ragas guide below shows how to do that.

We build RAG systems like this for companies as part of our AI consulting work, from preparing the documents to running the system in production.

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Sascha Gstir

About the author

Sascha Gstir

Co-Founder, Pondhouse Data

Sascha worked for almost a decade in data analytics and data science. At Pondhouse Data he builds and tunes AI solutions, from model selection to production.