Building RAG with Python

What is it?

RAG (retrieval-augmented generation) means: find relevant passages from your material, put them in the prompt, then ask the model to answer using that material. In Python you own every step as ordinary code — file reads, string splits, HTTP calls, and a place to keep vectors.

This page is a task guide, not a product ranking. Library names below describe roles (HTTP client, embedding helper, vector store). Use the standard library where it is enough; add a package when a role becomes painful.

Why it matters

Without retrieval, the model answers from parameters alone and may invent citations. With RAG you can:

• answer questions about private docs;
• update knowledge by re-indexing files instead of re-training;
• show which passages supported an answer.

Python is a common place to prototype this because scripts are easy to rerun while you tune Chunking: How to Split Documents for RAG and prompts.

How to do it

1. Load documents (UTF-8 text, or extract text from PDFs/HTML with a dedicated library).
2. Chunk into overlapping passages sized for your embedding model (Chunking: How to Split Documents for RAG).
3. Embed each chunk with an embedding API or local model (What are Embeddings in AI?).
4. Store vectors with metadata (source path, offsets). Start with a JSON file or in-memory list; move to a vector store or PostgreSQL when scale demands it.
5. Retrieve the top-k chunks for a new question (same embedding model).
6. Stuff the prompt with those chunks and a clear instruction: answer only from the passages; say when the answer is missing.
7. Check the answer against the passages (How to Reduce AI Hallucinations, How to Evaluate AI Systems and Outputs).

Keep the chat API call separate from retrieval so you can test each part (Calling AI and Model APIs with Python).

Example

Sketch of the pipeline (placeholders for provider-specific fields):

import json, math

def chunk_text(text, size=500, overlap=50):
    chunks, i = [], 0
    while i < len(text):
        chunks.append(text[i:i+size])
        i += max(1, size - overlap)
    return chunks

def cosine(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(y*y for y in b))
    return dot / (na * nb + 1e-9)

# embed(text) -> list[float]   # call your embedding HTTP API
# chat(prompt) -> str          # call your chat HTTP API

docs = chunk_text(open("handbook.txt", encoding="utf-8").read())
store = [{"text": c, "vec": embed(c)} for c in docs]

q = "How do we reset MFA?"
qv = embed(q)
top = sorted(store, key=lambda r: cosine(qv, r["vec"]), reverse=True)[:4]
context = "\n---\n".join(r["text"] for r in top)
prompt = (
    "Answer only using the passages. If missing, say you do not know.\n\n"
    f"Passages:\n{context}\n\nQuestion: {q}"
)
print(chat(prompt))

Replace embed and chat with thin wrappers around your provider. Persist store with JSON when the corpus grows past a toy example.

When to use it

Build RAG in Python when documents change faster than you can fine-tune, when answers must cite sources, or when you are evaluating retrieval quality offline. Prefer a hosted search product only after a local script proves the idea.

If the corpus is tiny and always fits in the context window, pasting the whole file may be simpler than retrieval — measure before you invest.

Common mistakes

• Chunks that are too large (diluted meaning) or too small (no context).
• Embedding with one model and querying with another.
• Stuffing retrieved text without telling the model to stick to it.
• Skipping an evaluation set of real questions (How to Evaluate AI Systems and Outputs).
• Putting API keys in the script body (Calling AI and Model APIs with Python).

Related concepts