Definition
Document summarization with LLMs condenses one or more long documents into a shorter text for a stated reader and purpose, usually by prompting an instruction-tuned model rather than training a summarizer. Documents longer than the context window are handled with multi-call strategies: map-reduce (summarize chunks, then summarize the summaries) and refine (update a running summary chunk by chunk). Wu et al. (2021) applied the recursive form to whole books.
The model internals and the classic extractive and abstractive models are covered in Text summarization. This article is about the applied problem: reports, contracts, transcripts and papers that are longer than a prompt, read by someone who needs specific things from them and will act on the result.
Three things make it hard. The document may not fit. Even when it fits, models use the middle of a long input less reliably than the start and end (Liu et al., 2023). And an abstractive summary can state things the source never said, which is worse in a summary than elsewhere because readers use it in place of the source.
Three strategies
| Strategy | Calls | Sequential steps | Strength | Weakness |
|---|---|---|---|---|
| Stuff | 1 | 1 | the model sees everything at once; cross-references stay intact | only works if L fits; the middle is under-used |
| Map-reduce | n + 1 (more if reduce overflows) | map calls run in parallel; each chunk is read closely | a chunk loses context defined elsewhere; reduce sees only partials | |
| Refine | n | n | each step sees the running summary, so later chunks are read in context | slow; each rewrite can drop earlier details, which biases the summary toward the end |
Cost and depth
- document length and chunk size in tokens
- instruction tokens per call and tokens per partial (or running) summary
- total input tokens read across all calls
- the model's context window
- fan-in: how many partial summaries one reduce call can read
- sequential rounds of map-reduce; refine always needs n
Both strategies read the document once, so their token totals are close. The difference is latency and what each call sees. Map-reduce runs its map calls in parallel and finishes in 2 rounds when the partials fit one reduce call. Refine has to wait for each call before starting the next.
Reference points
- Wu et al. (2021)
- OpenAI summarized whole books by recursively summarizing sections and then the summaries, with GPT-3 fine-tuned on human feedback
- Goyal, Li and Durrett (2022)
- human raters preferred zero-shot GPT-3 news summaries to those of fine-tuned summarizers, while ROUGE did not show this
- Chain of Density (2023)
- Adams et al.: repeatedly add missing salient entities at a fixed length; readers preferred summaries denser than a plain GPT-4 summary but not the densest
- Faithfulness metrics
- QAGS (Wang et al., 2020) asks questions of summary and source; SummaC (Laban et al., 2022) applies NLI sentence by sentence (see Hallucination and groundedness evaluation)
- Datasets
- CNN/DailyMail: about 300k news articles with highlight bullets; XSum (Narayan et al., 2018): 226,711 BBC articles with one-sentence summaries
Strategies in a simulation
The grid shows the two failure directions. Small chunks cut facts off from their context, so map-reduce at 4k keeps 7. Very large chunks bring back the lost-in-the-middle effect inside each map call, so 32k also keeps 7. Refine improves as chunks grow because fewer rewrites means less erosion of early facts.
Worked example: a 200k-token contract bundle
| Quantity | Map-reduce | Refine |
|---|---|---|
| Calls | 25 map + 1 reduce = 26 | 25 |
| Input tokens | 200,000 + 25 × 300 + (25 × 400 + 300) = 217,800 | 25 × 8,300 + 24 × 400 = 217,100 |
| Fan-in g | ⌊31,700 / 400⌋ = 79 | n/a |
| Sequential steps | 1 + ⌈log₇₉ 25⌉ = 2 | 25 |
| At L = 2,000,000 (n = 250) | 4 reduce calls, then 1: 3 steps | 250 steps |
The token bills differ by 0.3%. The wait does not: if a call takes about the same time either way, refine takes 25 call-times and map-reduce about 2. At 250 chunks the 250 partials (100,000 tokens) overflow one reduce call, so they are reduced in 4 groups of at most 79 and the 4 results in a final call.
1import re2from concurrent.futures import ThreadPoolExecutor34def n_tokens(text): # swap in the model's tokenizer5 return max(1, round(len(text) / 4))67def split_sections(doc, max_tok=8000):8 """Split on headings, then pack paragraphs so no chunk exceeds max_tok."""9 chunks = []10 for sec in re.split(r"(?m)^(?=#+ )", doc):11 cur = ""12 for para in sec.split("\n\n"):13 if cur and n_tokens(cur + para) > max_tok:14 chunks.append(cur)15 cur = ""16 cur += para + "\n\n"17 if cur.strip():18 chunks.append(cur)19 return chunks2021MAP = ("Summarize this part of a report for a CFO in at most 5 bullets. Keep figures, "22 "dates and owners exactly. End each bullet with its page in [brackets].\n\n{text}")23REDUCE = ("Merge these partial summaries into at most 8 bullets, most important first. "24 "Remove duplicates. Keep every [page] citation.\n\n{text}")2526def summarize(doc, llm, window=32000, chunk=8000):27 with ThreadPoolExecutor(max_workers=8) as pool: # map calls run in parallel28 parts = list(pool.map(lambda c: llm(MAP.format(text=c)), split_sections(doc, chunk)))29 while True: # reduce, recursively if needed30 joined = "\n".join(parts)31 if n_tokens(REDUCE.format(text=joined)) <= window:32 return llm(REDUCE.format(text=joined))33 groups, cur = [], []34 for p in parts: # pack partials into windows35 if cur and n_tokens(REDUCE.format(text="\n".join(cur + [p]))) > window:36 groups.append(cur)37 cur = []38 cur.append(p)39 if len(groups) + 1 >= len(parts): # no progress: partials too long40 raise ValueError("partial summaries do not fit; shorten the map output")41 parts = [llm(REDUCE.format(text="\n".join(g))) for g in groups + [cur]]4243# offline stand-in for the LLM: keeps the cited lines of whatever it is given44fake = lambda prompt: "\n".join(l for l in prompt.splitlines() if "[p." in l)45doc = "# Risks\n\nOne supplier provides 60% of parts [p.18]\n\n# MD&A\n\nRevenue rose 12% to $4.1B [p.41]"46print(summarize(doc, fake))47# One supplier provides 60% of parts [p.18]48# Revenue rose 12% to $4.1B [p.41]Check yourself
A 2,000,000-token archive is summarized with 8k chunks, 400-token partials, a 300-token instruction and a 32k window. How many sequential rounds does map-reduce need?
Check yourself
A refine-style summary of a long transcript mentions mostly what was said in the last hour. What is the most likely cause?
Remember
- ✓Stuff if the document fits; map-reduce or refine if it does not. Split on structure (sections, speakers), not fixed sizes, where you can.
- ✓Map-reduce: parallel calls and rounds; refine: sequential calls, similar token cost.
- ✓Small chunks lose context, large ones under-use their middle, and refine erodes early content.
- ✓Ask for citations in every partial and check claims against their spans before the reader sees the summary.
- ✓Specify reader, purpose, length and must-keep items; ROUGE is a weak guide to summary quality for LLM output.