Chunking
Splitting documents into retrievable pieces.
Breaking source material into pieces small enough to retrieve and fit in context. Chunk size and boundaries decide what can be found and how much surrounding meaning survives.
Chunk by meaning, not by character count. Splitting mid-argument produces fragments that retrieve well and answer badly. Overlap and metadata — title, section, date — usually matter more than a clever splitter.
The tension is unavoidable: small chunks retrieve precisely and lose their context, large chunks keep meaning and blur the match. The resolutions that work are contextual prefixes, overlap, or splitting on document structure — headings and sections — rather than on a fixed character count.
Splitting on structure, not punctuation
A 400-token fixed window cuts tables and procedure steps in half.
Split on headings and keep each procedure intact, then attach the heading path to every chunk. Retrieval quality improves immediately because a chunk that contains a whole instruction is answerable, while half an instruction is a trap.
- blogContextual retrieval in AI systems ↗
The practical hybrid-retrieval article: chunking, embeddings, BM25, rank fusion and reranking, plus prepending chunk context to reduce retrieval failures.
- paperLost in the middle: how language models use long contexts ↗
Measured U-shaped performance: relevant information in the middle of a long context is used less reliably than at the beginning or end. Order your context deliberately.
RAG · Retrieval-Augmented Generation
Look things up, then put what you found in the prompt.
Embedding · Vector
A list of numbers representing meaning, so similar things land close together.
Reranking · Reranker
A second pass that reorders candidates by actual relevance.
Progressive Disclosure
Reveal detail only when it becomes relevant.