Definition
Document chunking splits each source document into passages (chunks) that are embedded, indexed and retrieved as units. A chunker is defined by its boundaries (fixed token counts, separators, topic shifts or document structure), a maximum size and an overlap between neighbours. The original RAG paper (Lewis et al., 2020) used disjoint 100-word chunks of Wikipedia; production systems usually split on document structure first.
Chunking runs once per document at ingestion, between parsing (PDF, HTML, slides to text) and embedding (see RAG architecture). Every later stage inherits its choices. A chunk is the smallest thing retrieval can return, so if a boundary cuts an answer in two, no retriever can return the whole answer in one hit.
There are two opposing pressures. A chunk should be small enough that its single embedding is about one thing and that only relevant text enters the prompt. It should be large enough to contain a complete answer with the context needed to read it ("it", "the above table", "this setting").
Four ways to choose boundaries
| Strategy | Boundaries | Strength | Weakness |
|---|---|---|---|
| Fixed-size | every tokens, stride | simple, predictable sizes | ignores sentences and sections |
| Recursive | largest separator that fits: blank line, line, sentence, word | respects paragraphs; the common default | a heading can end up alone or glued to the wrong section |
| Semantic | where the cosine distance between adjacent sentence embeddings is high | topically coherent chunks | needs an embedding pass at ingestion; sizes vary |
| Structure-aware | headings, list items, table rows, code blocks, pages | keeps sections whole; headings add context | needs a parser per format (Markdown, HTML, PDF layout) |
Two refinements address the context problem without making chunks larger. Contextual chunk headers prepend the document title and heading path ("Returns policy > Damaged items: …") to each chunk before embedding. Small-to-big retrieval indexes small chunks but returns their parent section to the LLM.
Size and overlap
- document length, chunk size and overlap in tokens; consecutive chunks start S − O apart (the stride)
- number of fixed-size chunks; each extra chunk costs an embedding call and an index entry
- length of the answer span, , at a uniformly random position; with it is always inside some chunk
Overlap is insurance against boundaries, paid for in index size and in near-duplicate chunks that can crowd the top . Structure-aware boundaries reduce how much insurance you need, which is why recursive or heading-based splitting with modest overlap is a common default.
Measure sizes in the embedding model's tokens. Many splitters count characters by default, and an encoder silently truncates input beyond its limit: a 2,000-character chunk is about 500 English tokens, which a 256-token encoder never fully reads.
Reference points
- RAG and DPR (2020)
- disjoint 100-word passages from a December 2018 English Wikipedia dump, about 21M passages
- LangChain
RecursiveCharacterTextSplittertries separators["\n\n", "\n", " ", ""]in order and measures length in characters unless given a token-based length function- Encoder limits
- BERT-style encoders accept 512 tokens;
all-MiniLM-L6-v2truncates at 256 word pieces. Text past the limit is not embedded - OpenAI file search
- as documented in 2024, default chunks of 800 tokens with 400 tokens of overlap
- Contextual retrieval
- Anthropic (September 2024): prepending an LLM-written, chunk-specific context to each chunk before embedding and BM25 indexing; it reported 49% fewer failed top-20 retrievals, and 67% with reranking added
Worked example
| S | O | P(answer intact) | Chunks n | Duplication | Tokens embedded |
|---|---|---|---|---|---|
| 256 | 0 | 157/256 = 0.6133 | 391 | 1.00× | 100,096 |
| 256 | 64 | 157/192 = 0.8177 | 521 | 1.33× | 133,376 |
| 512 | 0 | 413/512 = 0.8066 | 196 | 1.00× | 100,352 |
| 512 | 64 | 413/448 = 0.9219 | 224 | 1.14× | 114,688 |
Doubling from 256 to 512 raises the intact probability by more than adding 64 tokens of overlap does, and adds no duplication. But each retrieved chunk now spends 512 prompt tokens to deliver a 100-token answer. With that is 2,560 context tokens per query instead of 1,280. Tokens embedded is , an upper bound, since the last chunk is shorter.
1import re23def n_tokens(text): # swap in the embedding model's tokenizer4 return len(text.split())56SEPS = (r"\n\s*\n", r"(?<=[.!?])\s+", r"\s+") # paragraph, sentence, word78def split_recursive(text, size, seps=SEPS):9 if n_tokens(text) <= size or not seps:10 return [text]11 out = []12 for part in re.split(seps[0], text):13 out += split_recursive(part, size, seps[1:])14 return [p for p in out if p.strip()]1516def pack(pieces, size, overlap):17 chunks, cur = [], []18 for p in pieces:19 if cur and n_tokens(" ".join(cur + [p])) > size:20 chunks.append(" ".join(cur))21 keep = [] # carry trailing pieces as overlap22 for q in reversed(cur):23 if n_tokens(" ".join([q] + keep)) > overlap:24 break25 keep.insert(0, q)26 cur = keep27 cur.append(p)28 return chunks + ([" ".join(cur)] if cur else [])2930def chunk_markdown(md, size=300, overlap=50):31 chunks, path = [], []32 for block in re.split(r"(?m)^(?=#{1,3} )", md):33 if not block.strip():34 continue35 head, _, body = block.partition("\n")36 level = len(head) - len(head.lstrip("#"))37 path = path[: level - 1] + [head.lstrip("# ").strip()]38 prefix = " > ".join(path) + ": "39 room = size - n_tokens(prefix)40 for c in pack(split_recursive(body.strip(), room), room, overlap):41 chunks.append({"text": prefix + c, "section": path[-1]})42 return chunks4344# chunk_markdown(doc, size=16, overlap=4) on the first two sections of the page in the diagram gives45# "Returns policy: Items can be returned within 30 days of delivery."46# "Returns policy: Refunds go to the original card."47# "Returns policy > Damaged items: Report damage within 48 h and attach a photo of the"48# "Returns policy > Damaged items: a photo of the box. We ship a replacement free."Check yourself
Fixed-size chunks of S = 512 tokens. How much overlap guarantees that every 80-token answer lies whole in at least one chunk?
Check yourself
Retrieval often returns the right section, but the LLM answers "the document does not say" because the chunk reads "Set it to 30 seconds" with no subject. What helps most?
Remember
- ✓A chunk is the unit of retrieval: if a boundary cuts the answer, no retriever can return it whole.
- ✓Recursive and structure-aware splitting respect paragraphs and sections; fixed-size splitting ignores them; semantic splitting follows topic shifts.
- ✓Fixed-size: , duplication , and keeps every -token answer whole.
- ✓Small chunks match precisely but lose context; large chunks keep context but dilute the embedding and cost prompt tokens.
- ✓Count tokens with the embedding model's tokenizer, prepend titles and heading paths, and tune size on recall@k.