# BLEU, ROUGE and METEOR

> BLEU (clipped n-gram precision × brevity penalty), ROUGE (n-gram and LCS recall) and METEOR (aligned unigram F with a fragmentation penalty) score generated text by surface overlap with references: cheap and reproducible, blind to paraphrase.

Source: https://www.mlfoundry.in/learn/bleu-rouge-and-meteor  
Section: Generative AI › Evaluation of Generative Models  
Level: core · 8 min read · updated 2026-10-21

**BLEU, ROUGE and METEOR:** Reference-based metrics that score a generated text by counting words and word sequences it shares with one or more human references. **BLEU** (Papineni, Roukos, Ward and Zhu, ACL 2002) was built for machine translation and is precision-oriented. **ROUGE** (Chin-Yew Lin, 2004) was built for summarisation and is recall-oriented. **METEOR** (Banerjee and Lavie, 2005) aligns unigrams, also through stems and synonyms, and weights recall above precision.

They answer a narrow question: how much of the reference wording does the output reproduce? That is useful when the output space is narrow and references are good, as in **Machine translation** and **Text summarization** regression tests. They need no model and no GPU, are deterministic, and run on millions of segments in seconds.

For open-ended generation they are weak. A correct answer phrased differently from the reference scores low, and a fluent wrong answer that reuses the reference's words scores high. Neural metrics such as **BERTScore**, and judges such as **LLM-as-a-judge**, address that, at the cost of a model dependency.

## BLEU

$$
p_n=\frac{\sum_{g\in\text{n-grams}(c)}\min\!\big(\text{count}_c(g),\ \max_{\text{ref}}\text{count}_{\text{ref}}(g)\big)}{\sum_{g}\text{count}_c(g)},\qquad \text{BLEU}=\text{BP}\cdot\exp\Big(\sum_{n=1}^{4}\tfrac14\log p_n\Big),\qquad \text{BP}=\begin{cases}1 & c>r\\ e^{\,1-r/c} & c\le r\end{cases}
$$

- $p_n$: modified (clipped) n-gram precision: a candidate n-gram counts at most as often as it appears in any single reference
- $c,\ r$: candidate length and effective reference length (the reference length closest to c)
- $\text{BP}$: brevity penalty: precision alone would reward very short outputs

BLEU is defined at the **corpus** level: clipped counts, totals and lengths are summed over all segments before the precisions and the penalty are computed. Averaging sentence-level BLEU gives a different, noisier number. For one sentence, any $p_n=0$ makes the geometric mean 0, so sentence BLEU needs smoothing; SacreBLEU's default "exp" smoothing replaces the $k$-th zero precision with $1/(2^k\cdot\text{total}_n)$.

## ROUGE and METEOR

$$
\text{ROUGE-N}_R=\frac{\#\text{ matched n-grams}}{\#\text{ n-grams in reference}},\qquad \text{ROUGE-L}:\ P=\frac{\text{LCS}}{|c|},\ R=\frac{\text{LCS}}{|r|},\ F=\frac{2PR}{P+R}
$$

- $\text{ROUGE-N}$: n-gram overlap, reported as recall in the original and usually as P, R and F1 by current packages
- $\text{LCS}$: length of the longest common subsequence: in-order matches, gaps allowed

$$
F_{\text{mean}}=\frac{10PR}{R+9P},\qquad \text{Pen}=0.5\Big(\frac{\text{chunks}}{m}\Big)^3,\qquad \text{METEOR}=F_{\text{mean}}\,(1-\text{Pen})
$$

- $m$: number of aligned unigrams; P = m/|c|, R = m/|r|
- $F_{\text{mean}}$: harmonic mean that weights recall 9 times as much as precision
- $\text{chunks}$: fewest runs of aligned words that are contiguous and in the same order in both sentences
- $\text{Pen}$: fragmentation penalty, at most 0.5 when every match is its own chunk

METEOR aligns in stages: exact surface forms first, then Porter stems, then WordNet synonyms, choosing the alignment with the fewest crossings. Later versions (METEOR 1.3, Meteor Universal) tune the constants per language and add paraphrase tables. NLTK's `meteor_score` uses the 2005 form with $\alpha=0.9,\ \beta=3,\ \gamma=0.5$, which is exactly the equation above.

> Figure: One candidate, one reference. BLEU clips n-gram matches to 5/6, 3/5, 1/4 and 0/3; the geometric mean with the smoothed p₄ = 1/6 is 0.380, the 6-vs-7-token brevity penalty is 0.846, and BLEU is 0.322 (0 without smoothing). ROUGE-1 and ROUGE-L both give F 0.769 because the LCS 'the cat on the mat' is the 5 matched unigrams in order. METEOR finds 5 exact matches in 2 chunks: 0.701. With the stem/synonym stage, sat aligns to sitting, m becomes 6 and METEOR rises to 0.853.

**Conventions that change the number**

- **Scale:** SacreBLEU and most papers report BLEU × 100; ROUGE is usually reported × 100 too
- **SacreBLEU:** Post (WMT 2018) showed that tokenisation and reference preprocessing shift BLEU by several points, and proposed one standard implementation with a version signature (tokenizer 13a by default, zh and ja-mecab for Chinese and Japanese)
- **ROUGE-L vs Lsum:** Google's rouge-score package computes ROUGE-L on the whole text and ROUGE-Lsum as a union LCS over sentences split on newlines; summarisation papers usually mean Lsum
- **Stemming:** rouge-score applies a Porter stemmer only if use_stemmer=True; results with and without differ
- **chrF:** Popović (2015): character n-gram F-score; less sensitive to tokenisation and morphology than BLEU, and a common companion metric in MT
- **Agreement with humans:** WMT metrics shared tasks have repeatedly found trained neural metrics agree with human judgments better than BLEU, especially between strong systems

## Try it

> Figure: Default: 'the cat sat on the mat' against two references, exp smoothing. BLEU 0.322 (32.16), ROUGE-1 0.769, ROUGE-2 0.545, ROUGE-L 0.769, METEOR 0.701. 'reordered' uses every reference word: ROUGE-1 1.000, but ROUGE-L falls to 0.571, BLEU to 0.595, and METEOR stays 0.988 because the words form only 2 chunks. 'the ×7' is clipped to 2 of 7 unigrams: BLEU 0.078 smoothed, 0 without. 'short' has p₁ = p₂ = 1 but BP = e^(1 − 7/2) = 0.082. 'paraphrase' scores BLEU 0.152 and METEOR 0.370; switch on the stem/synonym table and METEOR jumps to 0.701 while BLEU and ROUGE do not move.

Worked example: BLEU for candidate 'the cat sat on the mat', reference 'the cat is sitting on the mat' (c = 6, r = 7). Checked in Python

| Step | Values | Result |
| --- | --- | --- |
| unigrams | the×2, cat, on, mat match; sat does not | p₁ = 5/6 = 0.833 |
| bigrams | the cat, on the, the mat (not cat sat, sat on) | p₂ = 3/5 = 0.600 |
| trigrams | only on the mat | p₃ = 1/4 = 0.250 |
| 4-grams | none of 3; exp smoothing: 1/(2·3) | p₄ = 0.167 |
| geometric mean | exp(¼(ln 0.833 + ln 0.6 + ln 0.25 + ln 0.167)) | 0.380 |
| brevity penalty | c = 6 < r = 7: e^(1 − 7/6) | 0.846 |
| BLEU | 0.846 × 0.380 | 0.322, i.e. 32.16 |

One wrong verb form costs a third of the score: "sat" breaks two bigrams, two trigrams and all 4-grams. A human would rate this candidate as a near-perfect paraphrase.

```python
import sacrebleu
from rouge_score import rouge_scorer
from nltk.translate.meteor_score import meteor_score   # needs nltk.download("wordnet")

hyps = ["the cat sat on the mat"]
refs = [["the cat is sitting on the mat"]]               # one list per reference set

print(sacrebleu.corpus_bleu(hyps, refs))                 # BLEU = 32.16 ... plus a signature
print(sacrebleu.corpus_chrf(hyps, refs))

scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeLsum"], use_stemmer=True)
s = scorer.score(target=refs[0][0], prediction=hyps[0])
print({k: round(v.fmeasure, 3) for k, v in s.items()})  # rouge1 0.769, rouge2 0.545

print(meteor_score([refs[0][0].split()], hyps[0].split()))  # exact, Porter-stem and WordNet stages
```

Standard implementations (sacrebleu, rouge-score, nltk; not run here). Commented values are the hand computations above; sacrebleu's sentence_bleu uses exp smoothing by default

## Choosing among them

| Metric | Orientation | Matches | Order handling | Typical use |
| --- | --- | --- | --- | --- |
| BLEU | precision + brevity penalty | exact n-grams, n = 1–4 | via 2- to 4-grams | MT, corpus level |
| ROUGE-N | recall (F1 in packages) | exact n-grams, optional stemming | via n ≥ 2 | summarisation |
| ROUGE-L | LCS-based F | exact tokens | in-order subsequence, gaps free | summarisation |
| METEOR | recall-weighted F | exact, stem, synonym | chunk penalty | MT, captioning |

> **Never compare numbers from different setups:** BLEU from a different tokeniser, casing, number of references or implementation is a different metric. Report the SacreBLEU signature. Do not compare corpus BLEU with averaged sentence BLEU, ROUGE-L with ROUGE-Lsum, or scores with and without stemming. A 1-point BLEU difference on a few hundred segments is often within noise; use paired bootstrap resampling.

> **What the overlap metrics are still good for:** They are cheap regression tests. When a pipeline change drops ROUGE on a fixed summarisation set by several points, something changed, even if the metric cannot say whether for better. Pair them with a semantic metric (**BERTScore**), a factuality check (**Hallucination and groundedness evaluation**) and periodic **Human evaluation and arena-style ranking**.

- **Machine translation:** BLEU and chrF for system comparison on fixed test sets, alongside neural metrics
- **Summarisation:** ROUGE-1, ROUGE-2 and ROUGE-Lsum on news benchmarks such as CNN/DailyMail
- **Captioning:** BLEU-4 and METEOR are reported on COCO captions, next to CIDEr and SPICE
- **Regression tests:** detect output drift after a prompt, model or decoding change

**Question:** A candidate repeats 'the' seven times; the reference 'the cat is sitting on the mat' contains 'the' twice. What is its BLEU unigram precision p₁?

- A. 7/7 = 1.0
- B. 2/7 ≈ 0.286
- C. 1/7 ≈ 0.143
- D. 0

**Answer:** B. 2/7 ≈ 0.286 — Clipping caps each candidate n-gram at its maximum count in a single reference. 'the' appears twice in the reference, so 2 of the 7 candidate unigrams count. Without clipping, the degenerate output would get perfect unigram precision.

**Question:** 'on the mat the cat is sitting' uses exactly the words of the reference 'the cat is sitting on the mat'. Which score stays highest?

- A. ROUGE-L (LCS based)
- B. BLEU
- C. METEOR, because 7 matches form only 2 chunks
- D. ROUGE-2

**Answer:** C. METEOR, because 7 matches form only 2 chunks — All 7 unigrams match. METEOR's penalty is 0.5·(2/7)³ = 0.012, so it scores 0.988. ROUGE-L only gets the 4-word LCS ('the cat is sitting'): 0.571. BLEU loses higher-order n-grams at the swap: 0.595. ROUGE-2 is 0.833.

**Key points**

- BLEU = BP · geometric mean of clipped 1- to 4-gram precisions; computed over the whole corpus, smoothed for single sentences.
- ROUGE-N counts n-gram overlap (recall first); ROUGE-L uses the longest common subsequence, ROUGE-Lsum does it per sentence.
- METEOR: Fmean = 10PR/(R + 9P) over exact, stem and synonym alignments, times 1 − 0.5·(chunks/m)³.
- Scores depend on tokenisation, casing, stemming and reference count: use SacreBLEU and report the signature.
- Surface overlap misses paraphrase and rewards wrong answers with the right words; pair with semantic metrics and humans.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
