Chunking

The simplest strategy is a fixed-size sliding window: split the token stream every N tokens with an overlap of M tokens so that a sentence cut at a boundary appears whole in at least one chunk. Recursive splitting, the default in several RAG frameworks, tries a hierarchy of separators (section breaks, blank lines, sentence ends, then spaces) and only falls back to a finer separator when a piece is still too large. Structure-aware chunking uses the document's own markup: Markdown or HTML headings, the DOM, PDF layout analysis, and rules such as never splitting a table, a numbered list or a code block. Semantic chunking embeds consecutive sentences and starts a new chunk where the similarity between neighbours drops below a threshold. None of these is universally best; the right choice depends on document type and must be measured.

Hard constraints come from the embedding model and the context window. Many BERT-derived embedding models accept at most 512 tokens and silently truncate anything longer, so a chunk that looks fine in characters may lose its second half at embedding time; token counts must be computed with the embedding model's own tokenizer, not the generator's. Very small chunks embed precisely but lose surrounding context, while large chunks dilute the embedding with several topics and consume context budget. Parent-child or "small-to-big" retrieval resolves part of this tension by indexing small chunks for matching but passing the enclosing section to the model.

Two techniques address lost context directly. Late chunking (Günther et al., 2024) runs a long-context embedding model over the whole document first and then pools token embeddings per chunk, so each chunk vector is conditioned on the text around it. Contextual retrieval, published by Anthropic in September 2024, uses a language model to prepend a short, document-aware description to every chunk before embedding and BM25 indexing; in Anthropic's evaluation this reduced the top-20 retrieval failure rate by 49 %, and by 67 % when combined with reranking.

In practice each chunk should carry metadata: source document ID and version, title and heading path, page or anchor for citations, language, and the access-control labels of the source. Without them grounded answers cannot cite precisely, permission filtering cannot be applied at query time, and deleting or updating a document cannot remove its chunks. Common silent failures include PDF extraction that interleaves two columns or repeats headers and footers, pronouns such as "the fund" that lose their referent when a chunk is cut away from its heading, and overlap that returns near-duplicate chunks and wastes the context window. Any change of chunking strategy requires re-embedding the whole corpus, so strategies are best compared offline on a labelled question set using retrieval recall at k.