Definition
Text summarization maps a document (or a set of documents) to a much shorter text that keeps its most important content. Extractive systems choose sentences from . Abstractive systems generate token by token and may use words that are not in the source. Modern abstractive summarizers are pre-trained encoder–decoders fine-tuned on article–summary pairs: the pointer-generator network (See, Liu and Manning, 2017), BART (Lewis et al., 2020) and PEGASUS (Zhang, Zhao, Saleh and Liu, 2020).
The input is long (a CNN/DailyMail article averages about 780 tokens) and the output is short (about 56 tokens, 3–4 sentences). The model has to decide what matters, compress it, and stay faithful to the source. Only the last of these is easy to check, and even that is hard to automate.
Before deep learning, summarization was almost entirely extractive. Systems scored sentences by word frequency, position or graph centrality. TextRank (Mihalcea and Tarau, 2004) and LexRank (Erkan and Radev, 2004) run PageRank on a sentence-similarity graph. In news, simply taking the first three sentences (lead-3) is a strong baseline, because journalists put the key facts first.
Extractive: label sentences, then classify
Extractive training data does not exist directly, because human summaries are rewritten. The standard fix is a greedy oracle: add source sentences one at a time while ROUGE against the reference keeps rising, and label the chosen ones 1. A sentence encoder then learns these labels. BertSum (Liu and Lapata, 2019) puts a [CLS] token before every sentence of a BERT input and classifies each sentence's [CLS] vector:
- contextual vector of sentence (its [CLS] output), so each score sees the whole document
- oracle label: 1 if the greedy ROUGE oracle selected sentence i
- at test time, take the top 3 sentences by score, in document order
Extractive output is grammatical and faithful by construction. It cannot fuse two sentences, drop a clause or paraphrase, so it is long-winded on datasets whose references are highly abstractive.
Abstractive: generate, but copy when needed
Abstractive summarization is a sequence-to-sequence task (see Sequence-to-sequence (Seq2Seq) and Bahdanau attention). Early attention models (Rush, Chopra and Weston, 2015, on headline generation) had two failures on news. They could not output rare names outside the vocabulary, and they repeated themselves. See et al. fixed both with the pointer-generator network.
- attention over source positions at step , reused as a copy distribution
- context vector ; is the decoder state and the decoder input
- learned switch: near 1 to write from the vocabulary, near 0 to copy a source token
For repetition, See et al. keep a coverage vector , feed it into the attention, and add to the loss with . Attending again to a source word that has already been covered then costs loss.
Pre-training replaced these special mechanisms with scale. BART is pre-trained to reconstruct corrupted documents. PEGASUS uses gap-sentence generation: it removes the sentences with the highest ROUGE-1 against the rest of the document (30% of sentences) and trains the decoder to generate them. That objective is close to summarization itself. Both are large Transformer encoder–decoders, so they copy names through attention without an explicit pointer.
Benchmarks and reported ROUGE-1 / 2 / L (F1)
- CNN/DailyMail
- news with 3–4 bullet-point highlights (Hermann et al. 2015; non-anonymised version from See et al.): 287,226 / 13,368 / 11,490 train / val / test pairs. Mostly extractive references.
- XSum
- 226,711 BBC articles, each with a one-sentence summary written to answer “what is this article about?” (Narayan, Cohen and Lapata, 2018). Built so that copying sentences does not work.
- Lead-3 on CNN/DM
- 40.34 / 17.70 / 36.57, which beat the pointer-generator with coverage (39.53 / 17.28 / 36.38) on ROUGE
- BART-large
- CNN/DM 44.16 / 21.28 / 40.90; XSum 45.14 / 22.27 / 37.25
- PEGASUS-large (568M)
- pre-trained on HugeNews: CNN/DM 44.17 / 21.47 / 41.11; XSum 47.21 / 24.56 / 39.25
ROUGE: what the numbers mean
- count of n-gram in the candidate; the min clips repeated matches
- ROUGE-1 counts shared words, ROUGE-2 shared word pairs; precision divides by the candidate's n-gram count instead
- longest common subsequence: in-order matches that need not be contiguous
ROUGE (Lin, 2004) began as a recall measure. Papers now report F1, usually with Porter stemming. ROUGE-Lsum computes LCS sentence by sentence. Scores from different tokenizers and options are not comparable, so report the exact package and settings.
Worked example: a correct paraphrase scores low
Reference: "Boats were used to rescue 35 people". Candidate: "Rescuers used boats to save 35 people". Both have 7 tokens after lower-casing, so precision, recall and F1 coincide.
| Metric | Matches | Candidate / reference units | F1 |
|---|---|---|---|
| ROUGE-1 | used, boats, to, 35, people = 5 | 7 / 7 | 5/7 = 0.714 |
| ROUGE-2 | “35 people” = 1 | 6 / 6 | 1/6 = 0.167 |
| ROUGE-L | LCS “used to 35 people” = 4 | 7 / 7 | 4/7 = 0.571 |
The meaning is identical, yet ROUGE-2 is 0.167, because "boats were used" and "used boats" share no bigram. The reverse also happens: a summary can keep most n-grams and still swap who did what. ROUGE measures overlap, not correctness. Faithfulness needs separate checks (see Factuality evaluation and Hallucination detection).
1from transformers import pipeline2from rouge_score import rouge_scorer34summarize = pipeline("summarization", model="facebook/bart-large-cnn")5article = open("article.txt").read() # BART reads at most 1024 tokens; longer input is truncated6pred = summarize(article, max_length=142, min_length=56, num_beams=4,7 no_repeat_ngram_size=3, truncation=True)[0]["summary_text"]89reference = open("highlights.txt").read()10scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeLsum"], use_stemmer=True)11scores = scorer.score(reference, pred) # argument order: (target, prediction)12print({k: round(v.fmeasure, 4) for k, v in scores.items()})Use when
- +Extractive: faithful to the source and grammatical, cheap to train from oracle labels
- +Abstractive: compresses and paraphrases; one fine-tuned BART/PEGASUS handles news, dialogue and scientific text
- +Copy mechanisms (explicit, or implicit through attention) carry rare names into the summary
Avoid when
- −Extractive: redundant and verbose; cannot merge facts from two sentences
- −Abstractive: can hallucinate entities, numbers and relations
- −Input limits (1,024 tokens for BART) force truncation or chunking on long documents (see Long-context NLP)
- −ROUGE rewards surface overlap, so it misses paraphrase and factual errors
Check yourself
A pointer-generator has p_gen = 0.2, and the attention gives 0.5 to the source word "Okonjo", which is outside the vocabulary. What is P("Okonjo")?
Check yourself
Why does the extractive oracle use ROUGE against the human reference to create labels?
Remember
- ✓Extractive summarization selects source sentences (lead-3, TextRank, BertSum with greedy-oracle labels). Abstractive summarization generates new text.
- ✓Pointer-generator: , plus a coverage loss against repetition.
- ✓BART (denoising) and PEGASUS (gap-sentence generation) are pre-trained encoder–decoders that fine-tune into strong summarizers.
- ✓CNN/DailyMail is near-extractive (lead-3 scores 40.34 R-1). XSum has one-sentence, highly abstractive summaries.
- ✓ROUGE-N counts clipped n-gram matches and ROUGE-L uses the LCS. A correct paraphrase can get ROUGE-2 of 0.167, so check faithfulness separately.