What it is
LlamaIndex is a data framework for applications that answer from private documents. You connect loaders (PDF, web, Notion, databases), split text into nodes, embed those nodes, and store them in an index. A query engine retrieves relevant nodes and sends them to a model. That is retrieval-augmented generation with a particular API shape. The project was originally known as GPT Index; the name changed, the role did not.
It is not a chat product and not a model host. It overlaps LangChain, which can also build RAG, but LlamaIndex spends more of its API on indices, node parsing and query engines, while LangChain spends more on generic chains, tools and agents. Many teams use one, not both.
Why and when it is used
Choose LlamaIndex when the centre of the product is "ask questions of this corpus": a policy library, a wiki, a set of PDFs. The loaders and the index abstractions save you from writing glue on day one. Choose a direct implementation (your own chunker, your own vector store client, your own prompt) when you want every token of the prompt under your review and the corpus is simple. Choose LangChain or a plain agent loop when tool calling, not retrieval, is the main complexity.
Do not choose it because a diagram showed more boxes. Every box is a behaviour you must test: bad splits, missed tables, embeddings that do not match your language, and a model that ignores the context.
How it works
Documents enter through a reader and become nodes, chunks with metadata (source file, page, title). An embedding model turns each node into a vector. The index, often a vector index, stores those vectors. At query time the question is embedded, nearest nodes are fetched, optionally reranked, and written into a prompt. The model answers. Citations are only real if you pass source metadata through and require the answer to use it; the framework does not guarantee truth.
Advanced indices (lists of summaries, hierarchical chunks, knowledge graphs) try to handle corpora where a single vector lookup is not enough. They add calls and failure modes. Start with a vector index and a serious chunking scheme. Metadata filters (product, tenant, date) matter as much as the embedding model if you serve more than one customer.
Technologies and dependencies
Python is the primary library; a TypeScript port exists but the Python docs are the reference most examples use. You need an embedding model and a chat model, either via API keys or local weights. Storage can be the default in-memory index (fine for a demo, gone when the process exits) or a real vector store. Loaders may need extra system packages for PDF parsing. Other RAG frameworks and plain SQL with pgvector are alternatives once you outgrow the defaults.
How to get started
- Take ten documents you know well, not a dump of ten thousand.
- Ingest them, query them, and write down ten questions whose answers you can check by hand.
- Print the retrieved nodes for each question before you read the model's prose. If the right paragraph is missing, fix chunking or metadata. Do not tune the prompt yet.
- Add tenant or source metadata and a filter. Re-run the ten questions.
- Persist the index. Restart the process and confirm the index is still there.
- Only then consider a second index type or an agent on top.
Keep the evaluation questions in the repo. A framework upgrade that silently changes splitter defaults will show up there and nowhere else.
Cautions and trade-offs
Loaders are convenient and uneven. A PDF loader that drops tables will make a financial FAQ look "grounded" while missing the numbers. In-memory indices in production restart empty. Query engines hide the prompt; read it. Automatic "routing" across tools can retrieve nothing and still answer from the model's memory, which is a hallucination with extra steps.
LlamaIndex and LangChain both change APIs between major versions. Pin them. Do not send confidential documents to a hosted parsing service without reading where bytes go. And do not describe the system as "we use LlamaIndex" to a customer. Say what you index, how fresh it is, and what happens when retrieval misses.