EXPLAINER

Chunking: How to Split Documents for RAG

Chunking splits long documents into retrievable pieces. Chunk size and boundaries strongly affect whether retrieval finds the right information.

Short answer: Chunking divides documents into passages small enough to embed and retrieve precisely, but large enough to be self-explanatory. Split on natural structure (headings, paragraphs) first, and test sizes on real questions.

What is it?

Before documents go into a retrieval system (What is RAG (Retrieval-Augmented Generation)?), they are cut into pieces called chunks. Each chunk is embedded separately (What are Embeddings in AI?) and can be retrieved on its own. Chunking is one of the quietest but most influential choices in a RAG system: if the answer is split across two chunks, or buried inside a huge one, retrieval will struggle.

Why does it matter?

  • Chunks that are too large blend several topics, so their embedding is vague, and they waste space in the context window.
  • Chunks that are too small lose context ("It must be returned within 14 days" — what must?), and the right fragment may not rank highly.
  • Poor boundaries cut sentences, tables or code in half.

Many "the model gave a wrong answer" cases are really "retrieval fetched the wrong passage" cases.

How does it work?

Common strategies, from simple to richer:

  1. Fixed size — every N tokens or characters, often with some overlap between neighbours so sentences at the edges are not lost. Simple, but ignores structure.
  2. Structure-aware — split by headings, paragraphs, list items, or code functions, then merge small pieces up to a size limit. Usually a better default for documentation.
  3. Semantic — start a new chunk where the topic changes, detected by comparing neighbouring sentence embeddings. More effort; benefit varies.
  4. Hierarchical / parent-child — search small chunks for precision, but return the larger surrounding section to the model for context.

Practical tips:

  • Keep metadata with each chunk: document title, section heading, URL, date. Prepending the heading to the chunk text often helps retrieval.
  • Handle tables and code deliberately; do not split them mid-structure.
  • There is no universally best size. Typical starting points are a few hundred tokens, but treat that as a hypothesis to test.

Example

A returns policy with a heading "International orders" and three paragraphs. A fixed splitter might cut after the first paragraph's second sentence. A structure-aware splitter keeps the section together and stores title: "Returns Policy", section: "International orders". A question about "returns from Canada" now matches a chunk that actually says the rules for international orders.

To choose settings, assemble 20–50 real questions with known answer locations, try a few chunk sizes, and check how often the correct chunk appears in the top results (How to Evaluate AI Systems and Outputs).

When should I use it?

Any time you build retrieval over documents longer than a few paragraphs. For short, self-contained items (FAQ entries, product descriptions) the item itself is the chunk. Revisit chunking whenever retrieval answers look off.

Common mistakes

  • Copying a default without testing it.
  • Dropping headings and metadata, so chunks lose their context.
  • Huge overlap that bloats the index with near-duplicates.
  • Ignoring document cleaning — navigation menus, headers and footers repeated on every page pollute retrieval.
Learning path: Retrieval and grounding (RAG) (step 4 of 7)
← What is a Vector Database?What is a Context Window? →