Document chunking

Split documents into retrievable units before embedding: fixed, recursive, semantic or structure-aware. Size trades precise matching against context; overlap O keeps any answer of up to O + 1 tokens whole, at S/(S − O) index cost.

8 min read535 wordscore0 viewsupdated 2026-10-21

Definition

Document chunking splits each source document into passages (chunks) that are embedded, indexed and retrieved as units. A chunker is defined by its boundaries (fixed token counts, separators, topic shifts or document structure), a maximum size SS and an overlap OO between neighbours. The original RAG paper (Lewis et al., 2020) used disjoint 100-word chunks of Wikipedia; production systems usually split on document structure first.

Chunking runs once per document at ingestion, between parsing (PDF, HTML, slides to text) and embedding (see RAG architecture). Every later stage inherits its choices. A chunk is the smallest thing retrieval can return, so if a boundary cuts an answer in two, no retriever can return the whole answer in one hit.

There are two opposing pressures. A chunk should be small enough that its single embedding is about one thing and that only relevant text enters the prompt. It should be large enough to contain a complete answer with the context needed to read it ("it", "the above table", "this setting").

Four ways to choose boundaries

source document (markdown)# Returns policyItems can be returned within30 days of delivery. Refundsgo to the original card.## Damaged itemsReport damage within 48 hand attach a photo of thebox. We ship a replacementfree.## ExchangesExchanges are free for adifferent size or colour.fixed-size · every 40 tokens# Returns policyItems can be ret✂✂urned within 30days. Refunds go……## Damaged itemsReport damage wi✂cuts words and sentences, mixes sectionsrecursive · ¶ → sentence → word# Returns policyItems … card.## Damaged itemsReport … free.## ExchangesExchanges … colour.splits on the largest separator that fitssemantic · break at topic shiftsItems can be……original card.Report damage……replacement free.Exchanges are……or colour.breaks where neighbouring sentences divergestructure-aware · headings + path[Returns policy]Items … card.[Returns > Damaged]Report … free.[Returns > Exch.]Exchanges … colour.section boundaries, heading path prependedsize S = 512, overlap O = 64 on a 1,400-token documentchunk 1chunk 2chunk 304488961400stride S − O = 448 · chunks ⌈(1400 − 64)/448⌉ = 3 · overlap (red) is indexed twicelong-run duplication S/(S − O) = 1.14× · any answer of ≤ O + 1 = 65 tokens lies whole in one chunk
fig.The same three-section help page split four ways. Fixed-size chunks cut words and run across sections. A recursive splitter tries paragraphs first, then sentences, then words. A semantic splitter breaks where neighbouring sentences diverge in embedding space. A structure-aware splitter keeps sections whole and prepends the heading path. Below: size 512 with overlap 64 gives a stride of 448 and 1.14× duplication.
StrategyBoundariesStrengthWeakness
Fixed-sizeevery SS tokens, stride S−OS-Osimple, predictable sizesignores sentences and sections
Recursivelargest separator that fits: blank line, line, sentence, wordrespects paragraphs; the common defaulta heading can end up alone or glued to the wrong section
Semanticwhere the cosine distance between adjacent sentence embeddings is hightopically coherent chunksneeds an embedding pass at ingestion; sizes vary
Structure-awareheadings, list items, table rows, code blocks, pageskeeps sections whole; headings add contextneeds a parser per format (Markdown, HTML, PDF layout)
Boundary strategies

Two refinements address the context problem without making chunks larger. Contextual chunk headers prepend the document title and heading path ("Returns policy > Damaged items: …") to each chunk before embedding. Small-to-big retrieval indexes small chunks but returns their parent section to the LLM.

Size and overlap

n=⌈L−OS−O⌉,duplication≈SS−O,P(span of ℓ tokens intact)=min⁡ ⁣(1, S−ℓ+1S−O)n=\left\lceil\frac{L-O}{S-O}\right\rceil,\qquad \text{duplication}\approx\frac{S}{S-O},\qquad P(\text{span of }\ell\text{ tokens intact})=\min\!\Big(1,\ \frac{S-\ell+1}{S-O}\Big)
L, S, OL,\,S,\,O
document length, chunk size and overlap in tokens; consecutive chunks start S − O apart (the stride)
nn
number of fixed-size chunks; each extra chunk costs an embedding call and an index entry
ℓ\ell
length of the answer span, ℓ≤S\ell\le S, at a uniformly random position; with O≥ℓ−1O\ge\ell-1 it is always inside some chunk

Overlap is insurance against boundaries, paid for in index size and in near-duplicate chunks that can crowd the top kk. Structure-aware boundaries reduce how much insurance you need, which is why recursive or heading-based splitting with modest overlap is a common default.

Measure sizes in the embedding model's tokens. Many splitters count characters by default, and an encoder silently truncates input beyond its limit: a 2,000-character chunk is about 500 English tokens, which a 256-token encoder never fully reads.

Reference points

RAG and DPR (2020)
disjoint 100-word passages from a December 2018 English Wikipedia dump, about 21M passages
LangChain
RecursiveCharacterTextSplitter tries separators ["\n\n", "\n", " ", ""] in order and measures length in characters unless given a token-based length function
Encoder limits
BERT-style encoders accept 512 tokens; all-MiniLM-L6-v2 truncates at 256 word pieces. Text past the limit is not embedded
OpenAI file search
as documented in 2024, default chunks of 800 tokens with 400 tokens of overlap
Contextual retrieval
Anthropic (September 2024): prepending an LLM-written, chunk-specific context to each chunk before embedding and BM25 indexing; it reported 49% fewer failed top-20 retrievals, and 67% with reranking added
fixed-size · size 64 tokens · overlap 0 · 12 chunks732 tokens, 4 sections · bars = chunks (alternating lanes) · ■ answer intact ■ answer cut ■ heading | ¶061122183244305366427488549610671answers intact8 / 16 (50.0%)answer share of its chunk0.35chunks · mean size12 · 61.0 toktokens indexed / doc1.00×intact = one chunk holds every token of the answeranswers intact vs chunk size (fixed-size)0.00.51.032486496128192256
strategy
fig.A 732-token toy document with 16 answer spans, 4 of which cross a paragraph break. Default (fixed-size 64, no overlap): 8 of 16 answers survive whole and an answer fills 0.35 of its chunk. Overlap 16 lifts that to 12/16 at 1.31× indexed tokens. Recursive splitting at 64 also reaches 12/16 with no duplication, and structure-aware 13/16 for 1.08× (heading prefixes). Semantic at 96 keeps 15/16. Large chunks keep answers whole but dilute them: fixed-size 256 keeps 15/16 with an answer share of only 0.11.

Worked example

SOP(answer intact)Chunks nDuplicationTokens embedded
2560157/256 = 0.61333911.00×100,096
25664157/192 = 0.81775211.33×133,376
5120413/512 = 0.80661961.00×100,352
51264413/448 = 0.92192241.14×114,688
A 100,000-token corpus and a 100-token answer, fixed-size chunking (from the equations above)

Doubling SS from 256 to 512 raises the intact probability by more than adding 64 tokens of overlap does, and adds no duplication. But each retrieved chunk now spends 512 prompt tokens to deliver a 100-token answer. With k=5k=5 that is 2,560 context tokens per query instead of 1,280. Tokens embedded is n⋅Sn\cdot S, an upper bound, since the last chunk is shorter.

python
1import re
2
3def n_tokens(text): # swap in the embedding model's tokenizer
4 return len(text.split())
5
6SEPS = (r"\n\s*\n", r"(?<=[.!?])\s+", r"\s+") # paragraph, sentence, word
7
8def split_recursive(text, size, seps=SEPS):
9 if n_tokens(text) <= size or not seps:
10 return [text]
11 out = []
12 for part in re.split(seps[0], text):
13 out += split_recursive(part, size, seps[1:])
14 return [p for p in out if p.strip()]
15
16def pack(pieces, size, overlap):
17 chunks, cur = [], []
18 for p in pieces:
19 if cur and n_tokens(" ".join(cur + [p])) > size:
20 chunks.append(" ".join(cur))
21 keep = [] # carry trailing pieces as overlap
22 for q in reversed(cur):
23 if n_tokens(" ".join([q] + keep)) > overlap:
24 break
25 keep.insert(0, q)
26 cur = keep
27 cur.append(p)
28 return chunks + ([" ".join(cur)] if cur else [])
29
30def chunk_markdown(md, size=300, overlap=50):
31 chunks, path = [], []
32 for block in re.split(r"(?m)^(?=#{1,3} )", md):
33 if not block.strip():
34 continue
35 head, _, body = block.partition("\n")
36 level = len(head) - len(head.lstrip("#"))
37 path = path[: level - 1] + [head.lstrip("# ").strip()]
38 prefix = " > ".join(path) + ": "
39 room = size - n_tokens(prefix)
40 for c in pack(split_recursive(body.strip(), room), room, overlap):
41 chunks.append({"text": prefix + c, "section": path[-1]})
42 return chunks
43
44# chunk_markdown(doc, size=16, overlap=4) on the first two sections of the page in the diagram gives
45# "Returns policy: Items can be returned within 30 days of delivery."
46# "Returns policy: Refunds go to the original card."
47# "Returns policy > Damaged items: Report damage within 48 h and attach a photo of the"
48# "Returns policy > Damaged items: a photo of the box. We ship a replacement free."
A structure-aware chunker: split on headings, then recursively on paragraph, sentence and word, pack with overlap, and prepend the heading path (run here with whitespace tokens)

Check yourself

Fixed-size chunks of S = 512 tokens. How much overlap guarantees that every 80-token answer lies whole in at least one chunk?

Check yourself

Retrieval often returns the right section, but the LLM answers "the document does not say" because the chunk reads "Set it to 30 seconds" with no subject. What helps most?

Remember

  • ✓A chunk is the unit of retrieval: if a boundary cuts the answer, no retriever can return it whole.
  • ✓Recursive and structure-aware splitting respect paragraphs and sections; fixed-size splitting ignores them; semantic splitting follows topic shifts.
  • ✓Fixed-size: n=⌈(L−O)/(S−O)⌉n=\lceil (L-O)/(S-O)\rceil, duplication S/(S−O)S/(S-O), and O≥ℓ−1O\ge\ell-1 keeps every ℓ\ell-token answer whole.
  • ✓Small chunks match precisely but lose context; large chunks keep context but dilute the embedding and cost prompt tokens.
  • ✓Count tokens with the embedding model's tokenizer, prepend titles and heading paths, and tune size on recall@k.
← back to Generative AI roadmap