What it is
A RAG framework is a library that implements the moving parts of retrieval-augmented generation: reading files, splitting them, calling an embedding model, storing vectors, retrieving neighbours, and stuffing them into a prompt. LlamaIndex and LangChain are the names developers meet first. Others exist, including smaller libraries and vendor "assistants" products that hide the index entirely.
The framework is not the technique. You can build RAG with a database, an HTTP client and a prompt, which is what Building RAG with Python walks through. The framework is an accelerator and an opinion. It chooses defaults for splitter size, metadata and how the prompt is phrased. Those defaults are not neutral.
Why and when it is used
Use a framework when you are exploring a corpus and want loaders for messy formats tonight, or when your team would rather configure than write a vector-store client. Use plain code when the corpus is one format, the prompt must be reviewed line by line, or you have been burned by a major version upgrade. Use a managed retrieval product when you explicitly want to outsource indexing and can accept its data policy.
Do not adopt two frameworks at once. The overlap is confusing and the dependency tree is heavy. Do not adopt one before you can explain RAG without it. If you cannot say what a chunk is, the framework will only hide your confusion.
How it works
You configure an embedding model and a chat model, point a loader at a folder, and call an index or ingest function. The library chunks text, embeds it, and writes a store. A query function embeds the question, retrieves chunks, and calls the chat model with a prompt the library built. Some frameworks add rerankers, citation helpers and agent routers. Each extra stage is another model call with its own latency and failure.
What you should demand from any of them: a way to print the retrieved text, a way to persist the index, a way to filter by tenant, and a way to pin versions. If those are hard, the framework is a demo tool, not a production dependency.
Technologies and dependencies
Python or JavaScript, keys for the models you call, and a store (in-memory, Postgres with pgvector, or a dedicated vector database). PDF parsing may need native libraries. Observability is your job: the framework's trace product is optional and may send document text off-box. How to Choose an Agent Framework discusses the neighbouring decision when the system also calls tools.
How to get started
- Implement the smallest RAG path you can explain, even if it is the framework's quickstart.
- Freeze ten questions and the paragraphs that should be retrieved.
- Turn on whatever "show sources" debug the library has. Score retrieval separately from the final answer.
- Change chunk size once and re-run the ten. Keep the better setting.
- Restart the process and prove the index survived.
- Read the prompt template in the library source or docs and delete instructions you do not want.
If you cannot find the prompt, you do not control the product. Switch to your own template and call the model directly, using the framework only as a retriever if it still earns its place.
Cautions and trade-offs
Frameworks churn. Tutorials lie by omission when APIs move. Loaders drop content silently (scanned PDFs with no text, spreadsheets, headers). Default prompts tell the model to answer even when context is empty, which manufactures hallucinations. Multi-tenant bugs happen when a global index mixes customers.
A framework will not choose your chunking for legal documents versus chat logs. It will not know that "the policy" means last quarter's PDF, not the draft. Put that in metadata and filters. Evaluate retrieval hit-rate. And resist "agentic RAG" until the fixed pipeline is measurably good. An agent that searches again is not a fix for a bad index.