Document summarization

LLMs summarize documents longer than their context by stuffing, map-reduce or refine. Chunking decides what each call can see, the reduce or refine steps decide what survives, and citations plus claim checks keep the summary faithful.

8 min read540 wordscore0 viewsupdated 2026-10-21

Definition

Document summarization with LLMs condenses one or more long documents into a shorter text for a stated reader and purpose, usually by prompting an instruction-tuned model rather than training a summarizer. Documents longer than the context window are handled with multi-call strategies: map-reduce (summarize chunks, then summarize the summaries) and refine (update a running summary chunk by chunk). Wu et al. (2021) applied the recursive form to whole books.

The model internals and the classic extractive and abstractive models are covered in Text summarization. This article is about the applied problem: reports, contracts, transcripts and papers that are longer than a prompt, read by someone who needs specific things from them and will act on the result.

Three things make it hard. The document may not fit. Even when it fits, models use the middle of a long input less reliably than the start and end (Liu et al., 2023). And an abstractive summary can state things the source never said, which is worse in a summary than elsewhere because readers use it in place of the source.

Three strategies

map-reduce over a long report, with citations carried throughfictional annual report · 96k tokens · split on section headings, not fixed sizesannual report120 pages§1 business 9k§1A risks 22k§7 MD&A 18k§8 financials 31kother 16kmap prompt (each section)keep figures, dates, owners;cite [section page]§1A: one supplier gives60% of parts [1A p.18]§7: revenue +12% to$4.1B [7 p.41]§8: $1.2B debt duein 2026 [8 p.77]reducemerge, dedupe,rank by thereader's goal,≤ 250 words,keep citationsverifyeach claim vsits cited span(NLI or LLMjudge); fix ordrop failures5 partials × ~300 tokensfinal summary• Revenue rose 12% to $4.1B, mostly from services [§7 p.41]• $1.2B of debt matures in 2026; no refinancing is arranged yet [§8 p.77]• One supplier provides 60% of key components [§1A p.18]call shapesstuff: 1 call, input ≤ windowmap-reduce: n parallel + 1refine: n in sequencerunning summary passed on
fig.Map-reduce over a fictional 96k-token annual report. It is split on section headings, each section is summarized by the same map prompt with page citations, a reduce call merges and ranks the partials for the reader, and a verification pass checks every claim against its cited span before the final summary. Below: the call shapes of stuff (one call), map-reduce (n parallel calls, then one) and refine (n calls in sequence).
StrategyCallsSequential stepsStrengthWeakness
Stuff11the model sees everything at once; cross-references stay intactonly works if L fits; the middle is under-used
Map-reducen + 1 (more if reduce overflows)1+⌈log⁡gn⌉1+\lceil\log_g n\rceilmap calls run in parallel; each chunk is read closelya chunk loses context defined elsewhere; reduce sees only partials
Refinenneach step sees the running summary, so later chunks are read in contextslow; each rewrite can drop earlier details, which biases the summary toward the end
Strategies for documents of length L with chunks of C tokens

Cost and depth

n=⌈LC⌉,Tmap-reduce≈L+n p+(n s+p),Trefine≈n (C+p)+(n−1) sn=\Big\lceil\frac{L}{C}\Big\rceil,\qquad T_{\text{map-reduce}}\approx L+n\,p+(n\,s+p),\qquad T_{\text{refine}}\approx n\,(C+p)+(n-1)\,s
L,  CL,\;C
document length and chunk size in tokens
p,  sp,\;s
instruction tokens per call and tokens per partial (or running) summary
TT
total input tokens read across all calls
g=⌊W−ps⌋,depth=1+⌈log⁡gn⌉g=\Big\lfloor\frac{W-p}{s}\Big\rfloor,\qquad \text{depth}=1+\big\lceil\log_g n\big\rceil
WW
the model's context window
gg
fan-in: how many partial summaries one reduce call can read
depth\text{depth}
sequential rounds of map-reduce; refine always needs n

Both strategies read the document once, so their token totals are close. The difference is latency and what each call sees. Map-reduce runs its map calls in parallel and finishes in 2 rounds when the partials fit one reduce call. Refine has to wait for each call before starting the next.

Reference points

Wu et al. (2021)
OpenAI summarized whole books by recursively summarizing sections and then the summaries, with GPT-3 fine-tuned on human feedback
Goyal, Li and Durrett (2022)
human raters preferred zero-shot GPT-3 news summaries to those of fine-tuned summarizers, while ROUGE did not show this
Chain of Density (2023)
Adams et al.: repeatedly add missing salient entities at a fixed length; readers preferred summaries denser than a plain GPT-4 summary but not the densest
Faithfulness metrics
QAGS (Wang et al., 2020) asks questions of summary and source; SummaC (Laban et al., 2022) applies NLI sentence by sentence (see Hallucination and groundedness evaluation)
Datasets
CNN/DailyMail: about 300k news articles with highlight bullets; XSum (Narayan et al., 2018): 226,711 BBC articles with one-sentence summaries

Strategies in a simulation

map-reduce · 16k-token chunks · summary of 10 facts96k-token report, 30 facts (size = importance) · ○ ring = top-10 key fact · ● kept ⌒ needs earlier context0k32k64k96k6 map calls in parallel → 1 reduce call over 6 short summariesmapmapmapmapmapmapreducekey facts kept9 / 10LLM calls7sequential steps2input tokens read98,240key facts kept (of 10) with a 10-fact summary4k chunks8k chunks16k chunks32k chunksstuff5555map-reduce7797refine3468stuff ignores chunk size: it always reads the first 32k tokens
strategy
fig.A 96k-token report with 30 facts; the reference is the 10 most important. The simulated model keeps the facts it judges most important, under-uses the middle of long inputs, and under-rates facts whose context lies before the chunk it sees. Default (map-reduce, 16k chunks, 10-fact summary): 9 of 10 key facts in 7 calls and 2 steps, 98,240 tokens; the miss is the fact just after 64k whose context sits in the previous chunk. Stuffing a 32k window keeps 5 (the only key facts it can see). Refine keeps 3 with 4k chunks over 24 sequential calls, all from the second half, and 8 with 32k chunks.

The grid shows the two failure directions. Small chunks cut facts off from their context, so map-reduce at 4k keeps 7. Very large chunks bring back the lost-in-the-middle effect inside each map call, so 32k also keeps 7. Refine improves as chunks grow because fewer rewrites means less erosion of early facts.

Worked example: a 200k-token contract bundle

QuantityMap-reduceRefine
Calls25 map + 1 reduce = 2625
Input tokens200,000 + 25 × 300 + (25 × 400 + 300) = 217,80025 × 8,300 + 24 × 400 = 217,100
Fan-in g⌊31,700 / 400⌋ = 79n/a
Sequential steps1 + ⌈log₇₉ 25⌉ = 225
At L = 2,000,000 (n = 250)4 reduce calls, then 1: 3 steps250 steps
L = 200,000, C = 8,000, p = 300, s = 400, W = 32,000 (from the equations above)

The token bills differ by 0.3%. The wait does not: if a call takes about the same time either way, refine takes 25 call-times and map-reduce about 2. At 250 chunks the 250 partials (100,000 tokens) overflow one reduce call, so they are reduced in 4 groups of at most 79 and the 4 results in a final call.

python
1import re
2from concurrent.futures import ThreadPoolExecutor
3
4def n_tokens(text): # swap in the model's tokenizer
5 return max(1, round(len(text) / 4))
6
7def split_sections(doc, max_tok=8000):
8 """Split on headings, then pack paragraphs so no chunk exceeds max_tok."""
9 chunks = []
10 for sec in re.split(r"(?m)^(?=#+ )", doc):
11 cur = ""
12 for para in sec.split("\n\n"):
13 if cur and n_tokens(cur + para) > max_tok:
14 chunks.append(cur)
15 cur = ""
16 cur += para + "\n\n"
17 if cur.strip():
18 chunks.append(cur)
19 return chunks
20
21MAP = ("Summarize this part of a report for a CFO in at most 5 bullets. Keep figures, "
22 "dates and owners exactly. End each bullet with its page in [brackets].\n\n{text}")
23REDUCE = ("Merge these partial summaries into at most 8 bullets, most important first. "
24 "Remove duplicates. Keep every [page] citation.\n\n{text}")
25
26def summarize(doc, llm, window=32000, chunk=8000):
27 with ThreadPoolExecutor(max_workers=8) as pool: # map calls run in parallel
28 parts = list(pool.map(lambda c: llm(MAP.format(text=c)), split_sections(doc, chunk)))
29 while True: # reduce, recursively if needed
30 joined = "\n".join(parts)
31 if n_tokens(REDUCE.format(text=joined)) <= window:
32 return llm(REDUCE.format(text=joined))
33 groups, cur = [], []
34 for p in parts: # pack partials into windows
35 if cur and n_tokens(REDUCE.format(text="\n".join(cur + [p]))) > window:
36 groups.append(cur)
37 cur = []
38 cur.append(p)
39 if len(groups) + 1 >= len(parts): # no progress: partials too long
40 raise ValueError("partial summaries do not fit; shorten the map output")
41 parts = [llm(REDUCE.format(text="\n".join(g))) for g in groups + [cur]]
42
43# offline stand-in for the LLM: keeps the cited lines of whatever it is given
44fake = lambda prompt: "\n".join(l for l in prompt.splitlines() if "[p." in l)
45doc = "# Risks\n\nOne supplier provides 60% of parts [p.18]\n\n# MD&A\n\nRevenue rose 12% to $4.1B [p.41]"
46print(summarize(doc, fake))
47# One supplier provides 60% of parts [p.18]
48# Revenue rose 12% to $4.1B [p.41]
Map-reduce with heading-aware chunks, parallel map calls and recursive reduce (plain Python; run with an offline stand-in for the LLM)

Check yourself

A 2,000,000-token archive is summarized with 8k chunks, 400-token partials, a 300-token instruction and a 32k window. How many sequential rounds does map-reduce need?

Check yourself

A refine-style summary of a long transcript mentions mostly what was said in the last hour. What is the most likely cause?

Remember

  • ✓Stuff if the document fits; map-reduce or refine if it does not. Split on structure (sections, speakers), not fixed sizes, where you can.
  • ✓Map-reduce: n=⌈L/C⌉n=\lceil L/C\rceil parallel calls and 1+⌈log⁡gn⌉1+\lceil\log_g n\rceil rounds; refine: nn sequential calls, similar token cost.
  • ✓Small chunks lose context, large ones under-use their middle, and refine erodes early content.
  • ✓Ask for citations in every partial and check claims against their spans before the reader sees the summary.
  • ✓Specify reader, purpose, length and must-keep items; ROUGE is a weak guide to summary quality for LLM output.
← back to Generative AI roadmap