# Text summarization

> Summarization shortens a document while keeping its key content: extractive models select source sentences, abstractive ones (pointer-generator, BART, PEGASUS) write new text. Both are scored with ROUGE n-gram and LCS overlap.

Source: https://www.mlfoundry.in/learn/text-summarization  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 8 min read · updated 2026-10-15

**Text summarization:** maps a document $x$ (or a set of documents) to a much shorter text $y$ that keeps its most important content. **Extractive** systems choose sentences from $x$. **Abstractive** systems generate $y$ token by token and may use words that are not in the source. Modern abstractive summarizers are pre-trained encoder–decoders fine-tuned on article–summary pairs: the pointer-generator network (See, Liu and Manning, 2017), **BART** (Lewis et al., 2020) and PEGASUS (Zhang, Zhao, Saleh and Liu, 2020).

The input is long (a CNN/DailyMail article averages about 780 tokens) and the output is short (about 56 tokens, 3–4 sentences). The model has to decide what matters, compress it, and stay faithful to the source. Only the last of these is easy to check, and even that is hard to automate.

Before deep learning, summarization was almost entirely extractive. Systems scored sentences by word frequency, position or graph centrality. TextRank (Mihalcea and Tarau, 2004) and LexRank (Erkan and Radev, 2004) run PageRank on a sentence-similarity graph. In news, simply taking the first three sentences (**lead-3**) is a strong baseline, because journalists put the key facts first.

## Extractive: label sentences, then classify

Extractive training data does not exist directly, because human summaries are rewritten. The standard fix is a **greedy oracle**: add source sentences one at a time while ROUGE against the reference keeps rising, and label the chosen ones 1. A sentence encoder then learns these labels. BertSum (Liu and Lapata, 2019) puts a [CLS] token before every sentence of a BERT input and classifies each sentence's [CLS] vector:

$$
\hat y_i=\sigma\big(w^\top h_i+b\big),\qquad \mathcal L=-\sum_{i=1}^{n}\Big[y_i\log\hat y_i+(1-y_i)\log(1-\hat y_i)\Big]
$$

- $h_i$: contextual vector of sentence $i$ (its [CLS] output), so each score sees the whole document
- $y_i\in\{0,1\}$: oracle label: 1 if the greedy ROUGE oracle selected sentence i
- $\hat y_i$: at test time, take the top 3 sentences by score, in document order

Extractive output is grammatical and faithful by construction. It cannot fuse two sentences, drop a clause or paraphrase, so it is long-winded on datasets whose references are highly abstractive.

## Abstractive: generate, but copy when needed

Abstractive summarization is a sequence-to-sequence task (see **Sequence-to-sequence (Seq2Seq)** and **Bahdanau attention**). Early attention models (Rush, Chopra and Weston, 2015, on headline generation) had two failures on news. They could not output rare names outside the vocabulary, and they repeated themselves. See et al. fixed both with the pointer-generator network.

$$
P(w)=p_{\text{gen}}\,P_{\text{vocab}}(w)+\big(1-p_{\text{gen}}\big)\sum_{i:\,x_i=w}a_i^t,\qquad p_{\text{gen}}=\sigma\big(w_h^\top h_t^*+w_s^\top s_t+w_x^\top x_t+b\big)
$$

- $a^t$: attention over source positions at step $t$, reused as a copy distribution
- $h_t^*$: context vector $\sum_i a_i^t h_i$; $s_t$ is the decoder state and $x_t$ the decoder input
- $p_{\text{gen}}$: learned switch: near 1 to write from the vocabulary, near 0 to copy a source token

> Figure: One decoding step of a pointer-generator. The summary so far is "200 homes flooded in". The attention puts 0.62 on the source word Kirkby, which is not in the 50k-word vocabulary, so P_vocab gives it 0. With p_gen = 0.25 the final distribution still gives Kirkby 0.75 × 0.62 = 0.465. A word in both distributions, such as "in", gets 0.25 × 0.02 + 0.75 × 0.10 = 0.080. The coverage term along the bottom penalises attending again to source words already used.

For repetition, See et al. keep a **coverage vector** $c^t=\sum_{t'<t}a^{t'}$, feed it into the attention, and add $\lambda\sum_i\min(a_i^t,c_i^t)$ to the loss with $\lambda=1$. Attending again to a source word that has already been covered then costs loss.

Pre-training replaced these special mechanisms with scale. **BART** is pre-trained to reconstruct corrupted documents. PEGASUS uses **gap-sentence generation**: it removes the sentences with the highest ROUGE-1 against the rest of the document (30% of sentences) and trains the decoder to generate them. That objective is close to summarization itself. Both are large Transformer encoder–decoders, so they copy names through attention without an explicit pointer.

**Benchmarks and reported ROUGE-1 / 2 / L (F1)**

- **CNN/DailyMail:** news with 3–4 bullet-point highlights (Hermann et al. 2015; non-anonymised version from See et al.): 287,226 / 13,368 / 11,490 train / val / test pairs. Mostly extractive references.
- **XSum:** 226,711 BBC articles, each with a one-sentence summary written to answer “what is this article about?” (Narayan, Cohen and Lapata, 2018). Built so that copying sentences does not work.
- **Lead-3 on CNN/DM:** 40.34 / 17.70 / 36.57, which beat the pointer-generator with coverage (39.53 / 17.28 / 36.38) on ROUGE
- **BART-large:** CNN/DM 44.16 / 21.28 / 40.90; XSum 45.14 / 22.27 / 37.25
- **PEGASUS-large (568M):** pre-trained on HugeNews: CNN/DM 44.17 / 21.47 / 41.11; XSum 47.21 / 24.56 / 39.25

## ROUGE: what the numbers mean

$$
\text{R-}N_{\text{rec}}=\frac{\sum_{g\in\text{n-grams}(y^{\text{ref}})}\min\big(\#_{\hat y}(g),\#_{y^{\text{ref}}}(g)\big)}{\big|\text{n-grams}(y^{\text{ref}})\big|},\qquad \text{R-L: } P=\frac{\mathrm{LCS}}{|\hat y|},\ R=\frac{\mathrm{LCS}}{|y^{\text{ref}}|},\ F_1=\frac{2PR}{P+R}
$$

- $\#_{\hat y}(g)$: count of n-gram $g$ in the candidate; the min clips repeated matches
- $\text{R-}N$: ROUGE-1 counts shared words, ROUGE-2 shared word pairs; precision divides by the candidate's n-gram count instead
- $\mathrm{LCS}$: longest common subsequence: in-order matches that need not be contiguous

ROUGE (Lin, 2004) began as a recall measure. Papers now report F1, usually with Porter stemming. ROUGE-Lsum computes LCS sentence by sentence. Scores from different tokenizers and options are not comparable, so report the exact package and settings.

> Figure: An eight-sentence flood story scored against a three-sentence human reference (30 tokens). The default lead-3 summary gets ROUGE-1 / 2 / L F1 of 0.545 / 0.267 / 0.338. The greedy oracle picks S1, S3 and S7, giving 0.541 / 0.389 / 0.514. It roughly matches lead-3 on unigrams but wins clearly on bigrams and LCS because it includes the forecast sentence, and it stops at 3 even when k = 5. TextRank ranks S8 ("defences built after 2015") highest because it shares words with other sentences, and scores 0.457 / 0.294 / 0.371. Click sentences to see recall rise and precision fall as the summary grows.

## Worked example: a correct paraphrase scores low

Reference: "Boats were used to rescue 35 people". Candidate: "Rescuers used boats to save 35 people". Both have 7 tokens after lower-casing, so precision, recall and F1 coincide.

ROUGE of the paraphrase (computed in Python, no stemming)

| Metric | Matches | Candidate / reference units | F1 |
| --- | --- | --- | --- |
| ROUGE-1 | used, boats, to, 35, people = 5 | 7 / 7 | 5/7 = 0.714 |
| ROUGE-2 | “35 people” = 1 | 6 / 6 | 1/6 = 0.167 |
| ROUGE-L | LCS “used to 35 people” = 4 | 7 / 7 | 4/7 = 0.571 |

The meaning is identical, yet ROUGE-2 is 0.167, because "boats were used" and "used boats" share no bigram. The reverse also happens: a summary can keep most n-grams and still swap who did what. ROUGE measures overlap, not correctness. Faithfulness needs separate checks (see **Factuality evaluation** and **Hallucination detection**).

```python
from transformers import pipeline
from rouge_score import rouge_scorer

summarize = pipeline("summarization", model="facebook/bart-large-cnn")
article = open("article.txt").read()          # BART reads at most 1024 tokens; longer input is truncated
pred = summarize(article, max_length=142, min_length=56, num_beams=4,
                 no_repeat_ngram_size=3, truncation=True)[0]["summary_text"]

reference = open("highlights.txt").read()
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeLsum"], use_stemmer=True)
scores = scorer.score(reference, pred)        # argument order: (target, prediction)
print({k: round(v.fmeasure, 4) for k, v in scores.items()})
```

Abstractive summary with a fine-tuned BART checkpoint, then ROUGE with Google's rouge-score package (not run here)

> **Higher ROUGE can mean less faithful:** Abstractive models fine-tuned on XSum often add facts that are not in the article, because the XSum references themselves include outside information (Maynez et al., 2020, "On Faithfulness and Factuality in Abstractive Summarization"). ROUGE does not penalise this and may even reward it. Check entailment or QA-based consistency alongside ROUGE.

> **The dataset decides the method:** On CNN/DailyMail the references copy heavily from the first sentences. That is why lead-3 beat the pointer-generator on ROUGE and extractive models stay competitive. On XSum, extractive methods do poorly by design, and the gains come from abstractive pre-trained models. Always compare against lead-k and the extractive oracle for the dataset you use.

**Strengths**

- Extractive: faithful to the source and grammatical, cheap to train from oracle labels
- Abstractive: compresses and paraphrases; one fine-tuned BART/PEGASUS handles news, dialogue and scientific text
- Copy mechanisms (explicit, or implicit through attention) carry rare names into the summary

**Limitations**

- Extractive: redundant and verbose; cannot merge facts from two sentences
- Abstractive: can hallucinate entities, numbers and relations
- Input limits (1,024 tokens for BART) force truncation or chunking on long documents (see **Long-context NLP**)
- ROUGE rewards surface overlap, so it misses paraphrase and factual errors

**Question:** A pointer-generator has p_gen = 0.2, and the attention gives 0.5 to the source word "Okonjo", which is outside the vocabulary. What is P("Okonjo")?

- A. 0.1
- B. 0.4
- C. 0.5
- D. 0, because the word is out of vocabulary

**Answer:** B. 0.4 — $P_{\text{vocab}}(\text{Okonjo})=0$, so only the copy term counts: $(1-p_{\text{gen}})\cdot a=0.8\times0.5=0.4$. The copy distribution is defined over source positions, so out-of-vocabulary words can be produced.

**Question:** Why does the extractive oracle use ROUGE against the human reference to create labels?

- A. Human references are rewritten, so no source sentence is marked as 'in the summary'
- B. ROUGE is differentiable and can be trained directly
- C. It guarantees the extractive model will beat lead-3
- D. BERT requires binary labels for every token

**Answer:** A. Human references are rewritten, so no source sentence is marked as 'in the summary' — Datasets give article–abstract pairs, not sentence labels. The greedy oracle picks the source sentences that best reconstruct the reference by ROUGE and labels those 1, which turns summarization into per-sentence binary classification.

**Key points**

- Extractive summarization selects source sentences (lead-3, TextRank, BertSum with greedy-oracle labels). Abstractive summarization generates new text.
- Pointer-generator: $P(w)=p_{\text{gen}}P_{\text{vocab}}(w)+(1-p_{\text{gen}})\sum_{i:x_i=w}a_i^t$, plus a coverage loss against repetition.
- BART (denoising) and PEGASUS (gap-sentence generation) are pre-trained encoder–decoders that fine-tune into strong summarizers.
- CNN/DailyMail is near-extractive (lead-3 scores 40.34 R-1). XSum has one-sentence, highly abstractive summaries.
- ROUGE-N counts clipped n-gram matches and ROUGE-L uses the LCS. A correct paraphrase can get ROUGE-2 of 0.167, so check faithfulness separately.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
