BLEU, ROUGE and METEOR

BLEU (clipped n-gram precision × brevity penalty), ROUGE (n-gram and LCS recall) and METEOR (aligned unigram F with a fragmentation penalty) score generated text by surface overlap with references: cheap and reproducible, blind to paraphrase.

8 min read506 wordscore0 viewsupdated 2026-10-21

Definition

BLEU, ROUGE and METEOR Reference-based metrics that score a generated text by counting words and word sequences it shares with one or more human references. BLEU (Papineni, Roukos, Ward and Zhu, ACL 2002) was built for machine translation and is precision-oriented. ROUGE (Chin-Yew Lin, 2004) was built for summarisation and is recall-oriented. METEOR (Banerjee and Lavie, 2005) aligns unigrams, also through stems and synonyms, and weights recall above precision.

They answer a narrow question: how much of the reference wording does the output reproduce? That is useful when the output space is narrow and references are good, as in Machine translation and Text summarization regression tests. They need no model and no GPU, are deterministic, and run on millions of segments in seconds.

For open-ended generation they are weak. A correct answer phrased differently from the reference scores low, and a fluent wrong answer that reuses the reference's words scores high. Neural metrics such as BERTScore, and judges such as LLM-as-a-judge, address that, at the cost of a model dependency.

BLEU

pn=∑g∈n-grams(c)min⁡ ⁣(countc(g), max⁡refcountref(g))∑gcountc(g),BLEU=BP⋅exp⁡(∑n=1414log⁡pn),BP={1c>re 1−r/cc≤rp_n=\frac{\sum_{g\in\text{n-grams}(c)}\min\!\big(\text{count}_c(g),\ \max_{\text{ref}}\text{count}_{\text{ref}}(g)\big)}{\sum_{g}\text{count}_c(g)},\qquad \text{BLEU}=\text{BP}\cdot\exp\Big(\sum_{n=1}^{4}\tfrac14\log p_n\Big),\qquad \text{BP}=\begin{cases}1 & c>r\\ e^{\,1-r/c} & c\le r\end{cases}
pnp_n
modified (clipped) n-gram precision: a candidate n-gram counts at most as often as it appears in any single reference
c, rc,\ r
candidate length and effective reference length (the reference length closest to c)
BP\text{BP}
brevity penalty: precision alone would reward very short outputs

BLEU is defined at the corpus level: clipped counts, totals and lengths are summed over all segments before the precisions and the penalty are computed. Averaging sentence-level BLEU gives a different, noisier number. For one sentence, any pn=0p_n=0 makes the geometric mean 0, so sentence BLEU needs smoothing; SacreBLEU's default "exp" smoothing replaces the kk-th zero precision with 1/(2k⋅totaln)1/(2^k\cdot\text{total}_n).

ROUGE and METEOR

ROUGE-NR=# matched n-grams# n-grams in reference,ROUGE-L: P=LCS∣c∣, R=LCS∣r∣, F=2PRP+R\text{ROUGE-N}_R=\frac{\#\text{ matched n-grams}}{\#\text{ n-grams in reference}},\qquad \text{ROUGE-L}:\ P=\frac{\text{LCS}}{|c|},\ R=\frac{\text{LCS}}{|r|},\ F=\frac{2PR}{P+R}
ROUGE-N\text{ROUGE-N}
n-gram overlap, reported as recall in the original and usually as P, R and F1 by current packages
LCS\text{LCS}
length of the longest common subsequence: in-order matches, gaps allowed
Fmean=10PRR+9P,Pen=0.5(chunksm)3,METEOR=Fmean (1−Pen)F_{\text{mean}}=\frac{10PR}{R+9P},\qquad \text{Pen}=0.5\Big(\frac{\text{chunks}}{m}\Big)^3,\qquad \text{METEOR}=F_{\text{mean}}\,(1-\text{Pen})
mm
number of aligned unigrams; P = m/|c|, R = m/|r|
FmeanF_{\text{mean}}
harmonic mean that weights recall 9 times as much as precision
chunks\text{chunks}
fewest runs of aligned words that are contiguous and in the same order in both sentences
Pen\text{Pen}
fragmentation penalty, at most 0.5 when every match is its own chunk

METEOR aligns in stages: exact surface forms first, then Porter stems, then WordNet synonyms, choosing the alignment with the fewest crossings. Later versions (METEOR 1.3, Meteor Universal) tune the constants per language and add paraphrase tables. NLTK's meteor_score uses the 2005 form with α=0.9, β=3, γ=0.5\alpha=0.9,\ \beta=3,\ \gamma=0.5, which is exactly the equation above.

one candidate, one reference, three ways to count overlapreferencecandidatethecatissittingonthematthecatsatonthematsolid = exact match · dashed = stem/synonym stage only · c = 6, r = 7BLEU · clipped precisionp₁ = 5/6 = 0.833p₂ = 3/5 = 0.600p₃ = 1/4 = 0.250p₄ = 0/3 → smoothed 1/6BP = e^(1 − 7/6) = 0.846geo-mean(p) = 0.380BLEU = 0.322 (0 unsmoothed)ROUGE · recall-orientedR-1: 5 of 7 ref unigrams P .833 R .714 F .769R-2: 3 of 6 ref bigrams P .600 R .500 F .545R-L: LCS = 5 tokens 'the cat on the mat' P .833 R .714 F .769METEOR · alignmentexact: m = 5, chunks = 2P .833 R .714Fmean = 10PR/(R+9P) = .725pen = 0.5·(2/5)³ = .032score = .725·(1−.032) = .701+ sat≈sitting: m = 6, 2 chunks → .853BLEU = BP·exp(¼ Σ ln pₙ): corpus BLEU sums clipped counts and lengths over all sentences firstROUGE-L ignores the gap at "is sitting"; METEOR charges 0.5·(chunks/m)³ for broken order
fig.One candidate, one reference. BLEU clips n-gram matches to 5/6, 3/5, 1/4 and 0/3; the geometric mean with the smoothed p₄ = 1/6 is 0.380, the 6-vs-7-token brevity penalty is 0.846, and BLEU is 0.322 (0 without smoothing). ROUGE-1 and ROUGE-L both give F 0.769 because the LCS 'the cat on the mat' is the 5 matched unigrams in order. METEOR finds 5 exact matches in 2 chunks: 0.701. With the stem/synonym stage, sat aligns to sitting, m becomes 6 and METEOR rises to 0.853.

Conventions that change the number

Scale
SacreBLEU and most papers report BLEU × 100; ROUGE is usually reported × 100 too
SacreBLEU
Post (WMT 2018) showed that tokenisation and reference preprocessing shift BLEU by several points, and proposed one standard implementation with a version signature (tokenizer 13a by default, zh and ja-mecab for Chinese and Japanese)
ROUGE-L vs Lsum
Google's rouge-score package computes ROUGE-L on the whole text and ROUGE-Lsum as a union LCS over sentences split on newlines; summarisation papers usually mean Lsum
Stemming
rouge-score applies a Porter stemmer only if use_stemmer=True; results with and without differ
chrF
Popović (2015): character n-gram F-score; less sensitive to tokenisation and morphology than BLEU, and a common companion metric in MT
Agreement with humans
WMT metrics shared tasks have repeatedly found trained neural metrics agree with human judgments better than BLEU, especially between strong systems

Try it

ref 1: the cat is sitting on the matref 2: there is a cat on the matcandidate (6 tokens) · fill: clipped unigram match / no match · underline: METEOR exact / stem-synthecatsatonthematvertical bar = start of a METEOR chunk (contiguous run in both sentences), best ref = 1BLEU (N = 4, exp smoothing)p1 5/60.833p2 3/50.600p3 1/40.250p4 0/30.1670 matches → 1/(2^k·3)BP c=6 r=70.846exp(1 − r/c), candidate shorterBLEU0.322= BP · exp(Σ ln pₙ / 4) → 32.16 on the 0–100 scaleROUGE (F1, best ref) and METEORROUGE-10.769P 0.833 R 0.714ROUGE-20.545P 0.600 R 0.500ROUGE-L0.769LCS 5: P 0.833 R 0.714Fmean0.725m = 5: P 0.833 R 0.714penalty0.0320.5·(2 chunks / 5)³METEOR0.701= Fmean · (1 − penalty), Fmean = 10PR / (R + 9P)sentence-level scores on lower-cased whitespace tokens; METEOR stem-syn uses a toy table(sat/sits/sitting/rests → sit, feline → cat, rug → mat), not Porter + WordNet
candidate
referencesBLEU smoothingMETEOR match
fig.Default: 'the cat sat on the mat' against two references, exp smoothing. BLEU 0.322 (32.16), ROUGE-1 0.769, ROUGE-2 0.545, ROUGE-L 0.769, METEOR 0.701. 'reordered' uses every reference word: ROUGE-1 1.000, but ROUGE-L falls to 0.571, BLEU to 0.595, and METEOR stays 0.988 because the words form only 2 chunks. 'the ×7' is clipped to 2 of 7 unigrams: BLEU 0.078 smoothed, 0 without. 'short' has p₁ = p₂ = 1 but BP = e^(1 − 7/2) = 0.082. 'paraphrase' scores BLEU 0.152 and METEOR 0.370; switch on the stem/synonym table and METEOR jumps to 0.701 while BLEU and ROUGE do not move.
StepValuesResult
unigramsthe×2, cat, on, mat match; sat does notp₁ = 5/6 = 0.833
bigramsthe cat, on the, the mat (not cat sat, sat on)p₂ = 3/5 = 0.600
trigramsonly on the matp₃ = 1/4 = 0.250
4-gramsnone of 3; exp smoothing: 1/(2·3)p₄ = 0.167
geometric meanexp(¼(ln 0.833 + ln 0.6 + ln 0.25 + ln 0.167))0.380
brevity penaltyc = 6 < r = 7: e^(1 − 7/6)0.846
BLEU0.846 × 0.3800.322, i.e. 32.16
Worked example: BLEU for candidate 'the cat sat on the mat', reference 'the cat is sitting on the mat' (c = 6, r = 7). Checked in Python

One wrong verb form costs a third of the score: "sat" breaks two bigrams, two trigrams and all 4-grams. A human would rate this candidate as a near-perfect paraphrase.

python
1import sacrebleu
2from rouge_score import rouge_scorer
3from nltk.translate.meteor_score import meteor_score # needs nltk.download("wordnet")
4
5hyps = ["the cat sat on the mat"]
6refs = [["the cat is sitting on the mat"]] # one list per reference set
7
8print(sacrebleu.corpus_bleu(hyps, refs)) # BLEU = 32.16 ... plus a signature
9print(sacrebleu.corpus_chrf(hyps, refs))
10
11scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeLsum"], use_stemmer=True)
12s = scorer.score(target=refs[0][0], prediction=hyps[0])
13print({k: round(v.fmeasure, 3) for k, v in s.items()}) # rouge1 0.769, rouge2 0.545
14
15print(meteor_score([refs[0][0].split()], hyps[0].split())) # exact, Porter-stem and WordNet stages
Standard implementations (sacrebleu, rouge-score, nltk; not run here). Commented values are the hand computations above; sacrebleu's sentence_bleu uses exp smoothing by default

Choosing among them

MetricOrientationMatchesOrder handlingTypical use
BLEUprecision + brevity penaltyexact n-grams, n = 1–4via 2- to 4-gramsMT, corpus level
ROUGE-Nrecall (F1 in packages)exact n-grams, optional stemmingvia n ≥ 2summarisation
ROUGE-LLCS-based Fexact tokensin-order subsequence, gaps freesummarisation
METEORrecall-weighted Fexact, stem, synonymchunk penaltyMT, captioning

Where it's used

Machine translation

BLEU and chrF for system comparison on fixed test sets, alongside neural metrics

Summarisation

ROUGE-1, ROUGE-2 and ROUGE-Lsum on news benchmarks such as CNN/DailyMail

Captioning

BLEU-4 and METEOR are reported on COCO captions, next to CIDEr and SPICE

Regression tests

detect output drift after a prompt, model or decoding change

Check yourself

A candidate repeats 'the' seven times; the reference 'the cat is sitting on the mat' contains 'the' twice. What is its BLEU unigram precision p₁?

Check yourself

'on the mat the cat is sitting' uses exactly the words of the reference 'the cat is sitting on the mat'. Which score stays highest?

Remember

  • ✓BLEU = BP · geometric mean of clipped 1- to 4-gram precisions; computed over the whole corpus, smoothed for single sentences.
  • ✓ROUGE-N counts n-gram overlap (recall first); ROUGE-L uses the longest common subsequence, ROUGE-Lsum does it per sentence.
  • ✓METEOR: Fmean = 10PR/(R + 9P) over exact, stem and synonym alignments, times 1 − 0.5·(chunks/m)³.
  • ✓Scores depend on tokenisation, casing, stemming and reference count: use SacreBLEU and report the signature.
  • ✓Surface overlap misses paraphrase and rewards wrong answers with the right words; pair with semantic metrics and humans.
← back to Generative AI roadmap