What is RAG (Retrieval-Augmented Generation)?

What is it?

Retrieval-Augmented Generation (RAG) is a pattern, named in a 2020 research paper from Lewis et al., in which a system first retrieves relevant information from a knowledge source and then asks a language model to generate an answer using that information.

A plain LLM answers from what is stored in its parameters, which may be outdated, incomplete, or unaware of your private documents. RAG supplies the missing facts at the moment of the question.

Why does it matter?

RAG is the standard way to build "ask questions about our documents" features because it:

• uses current and private information without retraining a model;
• reduces (but does not eliminate) hallucinations by giving the model source text (AI Hallucinations: Why Models Make Things Up);
• allows citations — you know which passages were used;
• lets you update knowledge by changing documents, not the model;
• supports access control, because you decide which documents a given user's query may retrieve.

How does it work?

A RAG system has two phases.

Indexing (done ahead of time):

1. Collect and clean documents.
2. Split into chunks (Chunking: How to Split Documents for RAG).
3. Convert each chunk to an embedding (What are Embeddings in AI?).
4. Store vectors, text and metadata (What is a Vector Database? or another index).

Answering (every question):

1. Optionally rewrite the user's question into a better search query.
2. Retrieve the top matching chunks — by vector similarity, keyword search, or both (hybrid).
3. Optionally rerank the candidates with a more precise model.
4. Build a prompt: instructions + retrieved chunks + the question.
5. The model generates an answer, ideally with citations and a "not found" option.

// Sketch of the answering phase (pseudo-code, not a specific library)
const queryVec = await embed(question);
const hits = await store.search(queryVec, { topK: 5, filter: { allowed: user.groups } });
const prompt = buildPrompt({ instructions, sources: hits.map(h => h.text), question });
const answer = await llm.generate(prompt);   // "answer only from <sources>"

Two separate things can fail: retrieval (the right passage was not found) and generation (the model misused what it was given). Evaluate them separately (How to Evaluate AI Systems and Outputs).

Example

A company handbook assistant. Question: "How many days of parental leave do I get?"

Retrieval returns the chunks titled "Parental leave" and "Leave request process". The prompt tells the model to answer only from those excerpts and to cite the section. The reply quotes the policy and links the source. When someone asks about a benefit the handbook does not mention, the instructions make the model answer "I couldn't find this in the handbook" instead of guessing.

When should I use it?

Choose RAG when the problem is missing knowledge: private data, frequently changing information, large corpora, or a need for citations. If the problem is behaviour or style (a consistent format, tone, or specialised task), consider better prompting or What is Fine-Tuning an AI Model?. These are not exclusive; see RAG vs Fine-Tuning: Which Should You Use?.

If your whole corpus is small enough to fit comfortably in the context window, simply including it can be simpler than building retrieval — though cost and accuracy still deserve testing.

Common mistakes

• Assuming RAG guarantees truth. The model can still ignore, misread or over-extend the sources.
• Neglecting chunking and cleaning, then blaming the model.
• Retrieving too many passages, diluting the useful ones.
• No "not found" behaviour, so the model fills gaps with guesses.
• Skipping evaluation. Without a test set you cannot tell whether a change helped.
• Ignoring security: retrieved text is untrusted input and can contain instructions (What is Prompt Injection?).

Related concepts