# Document chunking

> Split documents into retrievable units before embedding: fixed, recursive, semantic or structure-aware. Size trades precise matching against context; overlap O keeps any answer of up to O + 1 tokens whole, at S/(S − O) index cost.

Source: https://www.mlfoundry.in/learn/document-chunking  
Section: Generative AI › Retrieval-Augmented Generation (RAG)  
Level: core · 8 min read · updated 2026-10-21

**Document chunking:** splits each source document into passages (chunks) that are embedded, indexed and retrieved as units. A chunker is defined by its **boundaries** (fixed token counts, separators, topic shifts or document structure), a maximum **size** $S$ and an **overlap** $O$ between neighbours. The original RAG paper (Lewis et al., 2020) used disjoint 100-word chunks of Wikipedia; production systems usually split on document structure first.

Chunking runs once per document at ingestion, between parsing (PDF, HTML, slides to text) and embedding (see **RAG architecture**). Every later stage inherits its choices. A chunk is the smallest thing retrieval can return, so if a boundary cuts an answer in two, no retriever can return the whole answer in one hit.

There are two opposing pressures. A chunk should be **small** enough that its single embedding is about one thing and that only relevant text enters the prompt. It should be **large** enough to contain a complete answer with the context needed to read it ("it", "the above table", "this setting").

## Four ways to choose boundaries

> Figure: The same three-section help page split four ways. Fixed-size chunks cut words and run across sections. A recursive splitter tries paragraphs first, then sentences, then words. A semantic splitter breaks where neighbouring sentences diverge in embedding space. A structure-aware splitter keeps sections whole and prepends the heading path. Below: size 512 with overlap 64 gives a stride of 448 and 1.14× duplication.

Boundary strategies

| Strategy | Boundaries | Strength | Weakness |
| --- | --- | --- | --- |
| Fixed-size | every $S$ tokens, stride $S-O$ | simple, predictable sizes | ignores sentences and sections |
| Recursive | largest separator that fits: blank line, line, sentence, word | respects paragraphs; the common default | a heading can end up alone or glued to the wrong section |
| Semantic | where the cosine distance between adjacent sentence embeddings is high | topically coherent chunks | needs an embedding pass at ingestion; sizes vary |
| Structure-aware | headings, list items, table rows, code blocks, pages | keeps sections whole; headings add context | needs a parser per format (Markdown, HTML, PDF layout) |

Two refinements address the context problem without making chunks larger. **Contextual chunk headers** prepend the document title and heading path ("Returns policy > Damaged items: …") to each chunk before embedding. **Small-to-big** retrieval indexes small chunks but returns their parent section to the LLM.

## Size and overlap

$$
n=\left\lceil\frac{L-O}{S-O}\right\rceil,\qquad \text{duplication}\approx\frac{S}{S-O},\qquad P(\text{span of }\ell\text{ tokens intact})=\min\!\Big(1,\ \frac{S-\ell+1}{S-O}\Big)
$$

- $L,\,S,\,O$: document length, chunk size and overlap in tokens; consecutive chunks start S − O apart (the stride)
- $n$: number of fixed-size chunks; each extra chunk costs an embedding call and an index entry
- $\ell$: length of the answer span, $\ell\le S$, at a uniformly random position; with $O\ge\ell-1$ it is always inside some chunk

Overlap is insurance against boundaries, paid for in index size and in near-duplicate chunks that can crowd the top $k$. Structure-aware boundaries reduce how much insurance you need, which is why recursive or heading-based splitting with modest overlap is a common default.

Measure sizes in the embedding model's tokens. Many splitters count characters by default, and an encoder silently truncates input beyond its limit: a 2,000-character chunk is about 500 English tokens, which a 256-token encoder never fully reads.

**Reference points**

- **RAG and DPR (2020):** disjoint 100-word passages from a December 2018 English Wikipedia dump, about 21M passages
- **LangChain:** `RecursiveCharacterTextSplitter` tries separators `["\n\n", "\n", " ", ""]` in order and measures length in characters unless given a token-based length function
- **Encoder limits:** BERT-style encoders accept 512 tokens; `all-MiniLM-L6-v2` truncates at 256 word pieces. Text past the limit is not embedded
- **OpenAI file search:** as documented in 2024, default chunks of 800 tokens with 400 tokens of overlap
- **Contextual retrieval:** Anthropic (September 2024): prepending an LLM-written, chunk-specific context to each chunk before embedding and BM25 indexing; it reported 49% fewer failed top-20 retrievals, and 67% with reranking added

> Figure: A 732-token toy document with 16 answer spans, 4 of which cross a paragraph break. Default (fixed-size 64, no overlap): 8 of 16 answers survive whole and an answer fills 0.35 of its chunk. Overlap 16 lifts that to 12/16 at 1.31× indexed tokens. Recursive splitting at 64 also reaches 12/16 with no duplication, and structure-aware 13/16 for 1.08× (heading prefixes). Semantic at 96 keeps 15/16. Large chunks keep answers whole but dilute them: fixed-size 256 keeps 15/16 with an answer share of only 0.11.

## Worked example

A 100,000-token corpus and a 100-token answer, fixed-size chunking (from the equations above)

| S | O | P(answer intact) | Chunks n | Duplication | Tokens embedded |
| --- | --- | --- | --- | --- | --- |
| 256 | 0 | 157/256 = 0.6133 | 391 | 1.00× | 100,096 |
| 256 | 64 | 157/192 = 0.8177 | 521 | 1.33× | 133,376 |
| 512 | 0 | 413/512 = 0.8066 | 196 | 1.00× | 100,352 |
| 512 | 64 | 413/448 = 0.9219 | 224 | 1.14× | 114,688 |

Doubling $S$ from 256 to 512 raises the intact probability by more than adding 64 tokens of overlap does, and adds no duplication. But each retrieved chunk now spends 512 prompt tokens to deliver a 100-token answer. With $k=5$ that is 2,560 context tokens per query instead of 1,280. Tokens embedded is $n\cdot S$, an upper bound, since the last chunk is shorter.

```python
import re

def n_tokens(text):                       # swap in the embedding model's tokenizer
    return len(text.split())

SEPS = (r"\n\s*\n", r"(?<=[.!?])\s+", r"\s+")   # paragraph, sentence, word

def split_recursive(text, size, seps=SEPS):
    if n_tokens(text) <= size or not seps:
        return [text]
    out = []
    for part in re.split(seps[0], text):
        out += split_recursive(part, size, seps[1:])
    return [p for p in out if p.strip()]

def pack(pieces, size, overlap):
    chunks, cur = [], []
    for p in pieces:
        if cur and n_tokens(" ".join(cur + [p])) > size:
            chunks.append(" ".join(cur))
            keep = []                      # carry trailing pieces as overlap
            for q in reversed(cur):
                if n_tokens(" ".join([q] + keep)) > overlap:
                    break
                keep.insert(0, q)
            cur = keep
        cur.append(p)
    return chunks + ([" ".join(cur)] if cur else [])

def chunk_markdown(md, size=300, overlap=50):
    chunks, path = [], []
    for block in re.split(r"(?m)^(?=#{1,3} )", md):
        if not block.strip():
            continue
        head, _, body = block.partition("\n")
        level = len(head) - len(head.lstrip("#"))
        path = path[: level - 1] + [head.lstrip("# ").strip()]
        prefix = " > ".join(path) + ": "
        room = size - n_tokens(prefix)
        for c in pack(split_recursive(body.strip(), room), room, overlap):
            chunks.append({"text": prefix + c, "section": path[-1]})
    return chunks

# chunk_markdown(doc, size=16, overlap=4) on the first two sections of the page in the diagram gives
# "Returns policy: Items can be returned within 30 days of delivery."
# "Returns policy: Refunds go to the original card."
# "Returns policy > Damaged items: Report damage within 48 h and attach a photo of the"
# "Returns policy > Damaged items: a photo of the box. We ship a replacement free."
```

A structure-aware chunker: split on headings, then recursively on paragraph, sentence and word, pack with overlap, and prepend the heading path (run here with whitespace tokens)

> **Parse before you chunk:** Most chunking failures are parsing failures: two-column PDFs read across columns, tables flattened into a stream of numbers, page headers and footers repeated inside every chunk. Look at a sample of your chunks as plain text before tuning $S$ and $O$; a better parser often beats any change in chunk size.

> **Tune on retrieval, not on intuition:** Chunk size interacts with the embedding model, the query style and $k$. Build a small set of (question, answer location) pairs, re-chunk with 2–4 settings, and compare recall@k and the prompt tokens per query (see **RAG evaluation**). Re-chunking means re-embedding, so do this before the index is large.

**Question:** Fixed-size chunks of S = 512 tokens. How much overlap guarantees that every 80-token answer lies whole in at least one chunk?

- A. 0
- B. 40
- C. 79
- D. 512

**Answer:** C. 79 — $P=\min(1,(S-\ell+1)/(S-O))$ reaches 1 when $S-O\le S-\ell+1$, i.e. $O\ge\ell-1=79$. The cost is a stride of 433 and duplication $512/433=1.18\times$.

**Question:** Retrieval often returns the right section, but the LLM answers "the document does not say" because the chunk reads "Set it to 30 seconds" with no subject. What helps most?

- A. Lower the overlap to zero
- B. Prepend the document title and heading path to each chunk, or return the parent section
- C. Switch to fixed-size chunks
- D. Raise the temperature

**Answer:** B. Prepend the document title and heading path to each chunk, or return the parent section — The chunk lacks context, not relevance. Contextual headers make both the embedding and the LLM's reading of the chunk refer to the right setting; small-to-big retrieval sends the whole section to the model while matching on the small chunk.

**Key points**

- A chunk is the unit of retrieval: if a boundary cuts the answer, no retriever can return it whole.
- Recursive and structure-aware splitting respect paragraphs and sections; fixed-size splitting ignores them; semantic splitting follows topic shifts.
- Fixed-size: $n=\lceil (L-O)/(S-O)\rceil$, duplication $S/(S-O)$, and $O\ge\ell-1$ keeps every $\ell$-token answer whole.
- Small chunks match precisely but lose context; large chunks keep context but dilute the embedding and cost prompt tokens.
- Count tokens with the embedding model's tokenizer, prepend titles and heading paths, and tune size on recall@k.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
