Text summarization

Summarization shortens a document while keeping its key content: extractive models select source sentences, abstractive ones (pointer-generator, BART, PEGASUS) write new text. Both are scored with ROUGE n-gram and LCS overlap.

8 min read721 wordscore0 viewsupdated 2026-10-15

Definition

Text summarization maps a document xx (or a set of documents) to a much shorter text yy that keeps its most important content. Extractive systems choose sentences from xx. Abstractive systems generate yy token by token and may use words that are not in the source. Modern abstractive summarizers are pre-trained encoder–decoders fine-tuned on article–summary pairs: the pointer-generator network (See, Liu and Manning, 2017), BART (Lewis et al., 2020) and PEGASUS (Zhang, Zhao, Saleh and Liu, 2020).

The input is long (a CNN/DailyMail article averages about 780 tokens) and the output is short (about 56 tokens, 3–4 sentences). The model has to decide what matters, compress it, and stay faithful to the source. Only the last of these is easy to check, and even that is hard to automate.

Before deep learning, summarization was almost entirely extractive. Systems scored sentences by word frequency, position or graph centrality. TextRank (Mihalcea and Tarau, 2004) and LexRank (Erkan and Radev, 2004) run PageRank on a sentence-similarity graph. In news, simply taking the first three sentences (lead-3) is a strong baseline, because journalists put the key facts first.

Extractive: label sentences, then classify

Extractive training data does not exist directly, because human summaries are rewritten. The standard fix is a greedy oracle: add source sentences one at a time while ROUGE against the reference keeps rising, and label the chosen ones 1. A sentence encoder then learns these labels. BertSum (Liu and Lapata, 2019) puts a [CLS] token before every sentence of a BERT input and classifies each sentence's [CLS] vector:

y^i=σ(w⊤hi+b),L=−∑i=1n[yilog⁡y^i+(1−yi)log⁡(1−y^i)]\hat y_i=\sigma\big(w^\top h_i+b\big),\qquad \mathcal L=-\sum_{i=1}^{n}\Big[y_i\log\hat y_i+(1-y_i)\log(1-\hat y_i)\Big]
hih_i
contextual vector of sentence ii (its [CLS] output), so each score sees the whole document
yi∈{0,1}y_i\in\{0,1\}
oracle label: 1 if the greedy ROUGE oracle selected sentence i
y^i\hat y_i
at test time, take the top 3 sentences by score, in document order

Extractive output is grammatical and faithful by construction. It cannot fuse two sentences, drop a clause or paraphrase, so it is long-winded on datasets whose references are highly abstractive.

Abstractive: generate, but copy when needed

Abstractive summarization is a sequence-to-sequence task (see Sequence-to-sequence (Seq2Seq) and Bahdanau attention). Early attention models (Rush, Chopra and Weston, 2015, on headline generation) had two failures on news. They could not output rare names outside the vocabulary, and they repeated themselves. See et al. fixed both with the pointer-generator network.

P(w)=pgen Pvocab(w)+(1−pgen)∑i: xi=wait,pgen=σ(wh⊤ht∗+ws⊤st+wx⊤xt+b)P(w)=p_{\text{gen}}\,P_{\text{vocab}}(w)+\big(1-p_{\text{gen}}\big)\sum_{i:\,x_i=w}a_i^t,\qquad p_{\text{gen}}=\sigma\big(w_h^\top h_t^*+w_s^\top s_t+w_x^\top x_t+b\big)
ata^t
attention over source positions at step tt, reused as a copy distribution
ht∗h_t^*
context vector ∑iaithi\sum_i a_i^t h_i; sts_t is the decoder state and xtx_t the decoder input
pgenp_{\text{gen}}
learned switch: near 1 to write from the vocabulary, near 0 to copy a source token
pointer-generator (See, Liu & Manning, 2017) · one decoding stepattention aᵗ over the source = the copy distribution0.050.070.080.100.62theLuneburstitsbanksandhit200homesinKirkbyencoder: 1-layer BiLSTM → hᵢ (256 units per direction)context h*ₜ = Σᵢ aᵢᵗ hᵢdecoder: LSTM state sₜ · eᵢᵗ = vᵀtanh(W_h hᵢ + W_s sₜ + b)<s>200homesfloodedin→ Kirkbysummary so far (teacher-forced in training)coverage cᵗ = Σ_(t′<t) aᵗ′; the loss adds λ·Σᵢ min(aᵢᵗ, cᵢᵗ) to stop re-attending (repetition)P_vocab = softmax(V′[sₜ; h*ₜ])fixed 50k-word vocabularythe0.21town0.17north0.08[UNK]0.06in0.02Kirkby: not in vocab → 0p_gen = σ(wₕᵀh*ₜ + wₛᵀsₜ+ wₓᵀxₜ + b) = 0.25×pP(w) = p·P_vocab(w)+ (1−p)·Σ_(i: wᵢ=w) aᵢᵗKirkby0.465copyin0.080boththe0.060bothhomes0.060copytown0.043generate
fig.One decoding step of a pointer-generator. The summary so far is "200 homes flooded in". The attention puts 0.62 on the source word Kirkby, which is not in the 50k-word vocabulary, so P_vocab gives it 0. With p_gen = 0.25 the final distribution still gives Kirkby 0.75 × 0.62 = 0.465. A word in both distributions, such as "in", gets 0.25 × 0.02 + 0.75 × 0.10 = 0.080. The coverage term along the bottom penalises attending again to source words already used.

For repetition, See et al. keep a coverage vector ct=∑t′<tat′c^t=\sum_{t'<t}a^{t'}, feed it into the attention, and add λ∑imin⁡(ait,cit)\lambda\sum_i\min(a_i^t,c_i^t) to the loss with λ=1\lambda=1. Attending again to a source word that has already been covered then costs loss.

Pre-training replaced these special mechanisms with scale. BART is pre-trained to reconstruct corrupted documents. PEGASUS uses gap-sentence generation: it removes the sentences with the highest ROUGE-1 against the rest of the document (30% of sentences) and trains the decoder to generate them. That objective is close to summarization itself. Both are large Transformer encoder–decoders, so they copy names through attention without an explicit pointer.

Benchmarks and reported ROUGE-1 / 2 / L (F1)

CNN/DailyMail
news with 3–4 bullet-point highlights (Hermann et al. 2015; non-anonymised version from See et al.): 287,226 / 13,368 / 11,490 train / val / test pairs. Mostly extractive references.
XSum
226,711 BBC articles, each with a one-sentence summary written to answer “what is this article about?” (Narayan, Cohen and Lapata, 2018). Built so that copying sentences does not work.
Lead-3 on CNN/DM
40.34 / 17.70 / 36.57, which beat the pointer-generator with coverage (39.53 / 17.28 / 36.38) on ROUGE
BART-large
CNN/DM 44.16 / 21.28 / 40.90; XSum 45.14 / 22.27 / 37.25
PEGASUS-large (568M)
pre-trained on HugeNews: CNN/DM 44.17 / 21.47 / 41.11; XSum 47.21 / 24.56 / 39.25

ROUGE: what the numbers mean

R-Nrec=∑g∈n-grams(yref)min⁡(#y^(g),#yref(g))∣n-grams(yref)∣,R-L: P=LCS∣y^∣, R=LCS∣yref∣, F1=2PRP+R\text{R-}N_{\text{rec}}=\frac{\sum_{g\in\text{n-grams}(y^{\text{ref}})}\min\big(\#_{\hat y}(g),\#_{y^{\text{ref}}}(g)\big)}{\big|\text{n-grams}(y^{\text{ref}})\big|},\qquad \text{R-L: } P=\frac{\mathrm{LCS}}{|\hat y|},\ R=\frac{\mathrm{LCS}}{|y^{\text{ref}}|},\ F_1=\frac{2PR}{P+R}
#y^(g)\#_{\hat y}(g)
count of n-gram gg in the candidate; the min clips repeated matches
R-N\text{R-}N
ROUGE-1 counts shared words, ROUGE-2 shared word pairs; precision divides by the candidate's n-gram count instead
LCS\mathrm{LCS}
longest common subsequence: in-order matches that need not be contiguous

ROUGE (Lin, 2004) began as a recall measure. Papers now report F1, usually with Porter stemming. ROUGE-Lsum computes LCS sentence by sentence. Scores from different tokenizers and options are not comparable, so report the exact package and settings.

lead-k, k = 3 · 3 sentences, 47 tokens · reference 30 tokensTextRankS1Heavy rain made the river Lune burst its banks on Sunday, flooding 200 homes in Kirkby.S2Rescue teams used boats to reach 35 people trapped upstairs, and no one was hurt.S3The Environment Agency issued a severe flood warning after 80 mm of rain in 24 hours.S4Schools will stay closed on Monday while engineers inspect the main bridge.S5Residents said the water rose faster than it did in the floods of 2015.S6The council has opened the town hall as a shelter with food and blankets.S7More rain is forecast for Tuesday, and river levels could rise again.S8A spokesperson said defences built after 2015 had protected the town centre.reference summary (green = word also in the selection):The river Lune flooded 200 homes in Kirkby after 80 mm of rain fell in a day.Boats were used to rescue 35 people.More rain is forecast for Tuesday.overlapPRF1ROUGE-121 match0.4470.7000.545ROUGE-210 match0.2170.3450.267ROUGE-L13 LCS0.2770.4330.338lead-3 vs oracle: R-1 F1 0.545 vs 0.541 · R-2 F1 0.267 vs 0.389tokens lower-cased, no stemming · P = overlap / candidate n-grams · R = overlap / reference n-grams
selector
click a sentence in the figure to add or remove it (switches to your pick)
fig.An eight-sentence flood story scored against a three-sentence human reference (30 tokens). The default lead-3 summary gets ROUGE-1 / 2 / L F1 of 0.545 / 0.267 / 0.338. The greedy oracle picks S1, S3 and S7, giving 0.541 / 0.389 / 0.514. It roughly matches lead-3 on unigrams but wins clearly on bigrams and LCS because it includes the forecast sentence, and it stops at 3 even when k = 5. TextRank ranks S8 ("defences built after 2015") highest because it shares words with other sentences, and scores 0.457 / 0.294 / 0.371. Click sentences to see recall rise and precision fall as the summary grows.

Worked example: a correct paraphrase scores low

Reference: "Boats were used to rescue 35 people". Candidate: "Rescuers used boats to save 35 people". Both have 7 tokens after lower-casing, so precision, recall and F1 coincide.

MetricMatchesCandidate / reference unitsF1
ROUGE-1used, boats, to, 35, people = 57 / 75/7 = 0.714
ROUGE-2“35 people” = 16 / 61/6 = 0.167
ROUGE-LLCS “used to 35 people” = 47 / 74/7 = 0.571
ROUGE of the paraphrase (computed in Python, no stemming)

The meaning is identical, yet ROUGE-2 is 0.167, because "boats were used" and "used boats" share no bigram. The reverse also happens: a summary can keep most n-grams and still swap who did what. ROUGE measures overlap, not correctness. Faithfulness needs separate checks (see Factuality evaluation and Hallucination detection).

python
1from transformers import pipeline
2from rouge_score import rouge_scorer
3
4summarize = pipeline("summarization", model="facebook/bart-large-cnn")
5article = open("article.txt").read() # BART reads at most 1024 tokens; longer input is truncated
6pred = summarize(article, max_length=142, min_length=56, num_beams=4,
7 no_repeat_ngram_size=3, truncation=True)[0]["summary_text"]
8
9reference = open("highlights.txt").read()
10scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeLsum"], use_stemmer=True)
11scores = scorer.score(reference, pred) # argument order: (target, prediction)
12print({k: round(v.fmeasure, 4) for k, v in scores.items()})
Abstractive summary with a fine-tuned BART checkpoint, then ROUGE with Google's rouge-score package (not run here)

Use when

  • +Extractive: faithful to the source and grammatical, cheap to train from oracle labels
  • +Abstractive: compresses and paraphrases; one fine-tuned BART/PEGASUS handles news, dialogue and scientific text
  • +Copy mechanisms (explicit, or implicit through attention) carry rare names into the summary

Avoid when

  • −Extractive: redundant and verbose; cannot merge facts from two sentences
  • −Abstractive: can hallucinate entities, numbers and relations
  • −Input limits (1,024 tokens for BART) force truncation or chunking on long documents (see Long-context NLP)
  • −ROUGE rewards surface overlap, so it misses paraphrase and factual errors

Check yourself

A pointer-generator has p_gen = 0.2, and the attention gives 0.5 to the source word "Okonjo", which is outside the vocabulary. What is P("Okonjo")?

Check yourself

Why does the extractive oracle use ROUGE against the human reference to create labels?

Remember

  • ✓Extractive summarization selects source sentences (lead-3, TextRank, BertSum with greedy-oracle labels). Abstractive summarization generates new text.
  • ✓Pointer-generator: P(w)=pgenPvocab(w)+(1−pgen)∑i:xi=waitP(w)=p_{\text{gen}}P_{\text{vocab}}(w)+(1-p_{\text{gen}})\sum_{i:x_i=w}a_i^t, plus a coverage loss against repetition.
  • ✓BART (denoising) and PEGASUS (gap-sentence generation) are pre-trained encoder–decoders that fine-tune into strong summarizers.
  • ✓CNN/DailyMail is near-extractive (lead-3 scores 40.34 R-1). XSum has one-sentence, highly abstractive summaries.
  • ✓ROUGE-N counts clipped n-gram matches and ROUGE-L uses the LCS. A correct paraphrase can get ROUGE-2 of 0.167, so check faithfulness separately.
← back to Deep Learning roadmap