Definition
BLEU, ROUGE and METEOR Reference-based metrics that score a generated text by counting words and word sequences it shares with one or more human references. BLEU (Papineni, Roukos, Ward and Zhu, ACL 2002) was built for machine translation and is precision-oriented. ROUGE (Chin-Yew Lin, 2004) was built for summarisation and is recall-oriented. METEOR (Banerjee and Lavie, 2005) aligns unigrams, also through stems and synonyms, and weights recall above precision.
They answer a narrow question: how much of the reference wording does the output reproduce? That is useful when the output space is narrow and references are good, as in Machine translation and Text summarization regression tests. They need no model and no GPU, are deterministic, and run on millions of segments in seconds.
For open-ended generation they are weak. A correct answer phrased differently from the reference scores low, and a fluent wrong answer that reuses the reference's words scores high. Neural metrics such as BERTScore, and judges such as LLM-as-a-judge, address that, at the cost of a model dependency.
BLEU
- modified (clipped) n-gram precision: a candidate n-gram counts at most as often as it appears in any single reference
- candidate length and effective reference length (the reference length closest to c)
- brevity penalty: precision alone would reward very short outputs
BLEU is defined at the corpus level: clipped counts, totals and lengths are summed over all segments before the precisions and the penalty are computed. Averaging sentence-level BLEU gives a different, noisier number. For one sentence, any makes the geometric mean 0, so sentence BLEU needs smoothing; SacreBLEU's default "exp" smoothing replaces the -th zero precision with .
ROUGE and METEOR
- n-gram overlap, reported as recall in the original and usually as P, R and F1 by current packages
- length of the longest common subsequence: in-order matches, gaps allowed
- number of aligned unigrams; P = m/|c|, R = m/|r|
- harmonic mean that weights recall 9 times as much as precision
- fewest runs of aligned words that are contiguous and in the same order in both sentences
- fragmentation penalty, at most 0.5 when every match is its own chunk
METEOR aligns in stages: exact surface forms first, then Porter stems, then WordNet synonyms, choosing the alignment with the fewest crossings. Later versions (METEOR 1.3, Meteor Universal) tune the constants per language and add paraphrase tables. NLTK's meteor_score uses the 2005 form with , which is exactly the equation above.
Conventions that change the number
- Scale
- SacreBLEU and most papers report BLEU × 100; ROUGE is usually reported × 100 too
- SacreBLEU
- Post (WMT 2018) showed that tokenisation and reference preprocessing shift BLEU by several points, and proposed one standard implementation with a version signature (tokenizer 13a by default, zh and ja-mecab for Chinese and Japanese)
- ROUGE-L vs Lsum
- Google's rouge-score package computes ROUGE-L on the whole text and ROUGE-Lsum as a union LCS over sentences split on newlines; summarisation papers usually mean Lsum
- Stemming
- rouge-score applies a Porter stemmer only if use_stemmer=True; results with and without differ
- chrF
- Popović (2015): character n-gram F-score; less sensitive to tokenisation and morphology than BLEU, and a common companion metric in MT
- Agreement with humans
- WMT metrics shared tasks have repeatedly found trained neural metrics agree with human judgments better than BLEU, especially between strong systems
Try it
| Step | Values | Result |
|---|---|---|
| unigrams | the×2, cat, on, mat match; sat does not | p₁ = 5/6 = 0.833 |
| bigrams | the cat, on the, the mat (not cat sat, sat on) | p₂ = 3/5 = 0.600 |
| trigrams | only on the mat | p₃ = 1/4 = 0.250 |
| 4-grams | none of 3; exp smoothing: 1/(2·3) | p₄ = 0.167 |
| geometric mean | exp(¼(ln 0.833 + ln 0.6 + ln 0.25 + ln 0.167)) | 0.380 |
| brevity penalty | c = 6 < r = 7: e^(1 − 7/6) | 0.846 |
| BLEU | 0.846 × 0.380 | 0.322, i.e. 32.16 |
One wrong verb form costs a third of the score: "sat" breaks two bigrams, two trigrams and all 4-grams. A human would rate this candidate as a near-perfect paraphrase.
1import sacrebleu2from rouge_score import rouge_scorer3from nltk.translate.meteor_score import meteor_score # needs nltk.download("wordnet")45hyps = ["the cat sat on the mat"]6refs = [["the cat is sitting on the mat"]] # one list per reference set78print(sacrebleu.corpus_bleu(hyps, refs)) # BLEU = 32.16 ... plus a signature9print(sacrebleu.corpus_chrf(hyps, refs))1011scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeLsum"], use_stemmer=True)12s = scorer.score(target=refs[0][0], prediction=hyps[0])13print({k: round(v.fmeasure, 3) for k, v in s.items()}) # rouge1 0.769, rouge2 0.5451415print(meteor_score([refs[0][0].split()], hyps[0].split())) # exact, Porter-stem and WordNet stagesChoosing among them
| Metric | Orientation | Matches | Order handling | Typical use |
|---|---|---|---|---|
| BLEU | precision + brevity penalty | exact n-grams, n = 1–4 | via 2- to 4-grams | MT, corpus level |
| ROUGE-N | recall (F1 in packages) | exact n-grams, optional stemming | via n ≥ 2 | summarisation |
| ROUGE-L | LCS-based F | exact tokens | in-order subsequence, gaps free | summarisation |
| METEOR | recall-weighted F | exact, stem, synonym | chunk penalty | MT, captioning |
Where it's used
Machine translation
BLEU and chrF for system comparison on fixed test sets, alongside neural metrics
Summarisation
ROUGE-1, ROUGE-2 and ROUGE-Lsum on news benchmarks such as CNN/DailyMail
Captioning
BLEU-4 and METEOR are reported on COCO captions, next to CIDEr and SPICE
Regression tests
detect output drift after a prompt, model or decoding change
Check yourself
A candidate repeats 'the' seven times; the reference 'the cat is sitting on the mat' contains 'the' twice. What is its BLEU unigram precision p₁?
Check yourself
'on the mat the cat is sitting' uses exactly the words of the reference 'the cat is sitting on the mat'. Which score stays highest?
Remember
- ✓BLEU = BP · geometric mean of clipped 1- to 4-gram precisions; computed over the whole corpus, smoothed for single sentences.
- ✓ROUGE-N counts n-gram overlap (recall first); ROUGE-L uses the longest common subsequence, ROUGE-Lsum does it per sentence.
- ✓METEOR: Fmean = 10PR/(R + 9P) over exact, stem and synonym alignments, times 1 − 0.5·(chunks/m)³.
- ✓Scores depend on tokenisation, casing, stemming and reference count: use SacreBLEU and report the signature.
- ✓Surface overlap misses paraphrase and rewards wrong answers with the right words; pair with semantic metrics and humans.