BERTScore

BERTScore matches every token of candidate and reference to its most similar token in the other by cosine of contextual embeddings; mean best matches give precision, recall and F1, so paraphrases score high where n-gram overlap fails.

7 min read629 wordscore0 viewsupdated 2026-10-21

Definition

BERTScore A reference-based text-generation metric that compares contextual token embeddings instead of surface strings. Each token is matched greedily to its most similar token in the other text; the averages give precision, recall and F1. Introduced by Zhang, Kishore, Wu, Weinberger and Artzi, "BERTScore: Evaluating Text Generation with BERT" (ICLR 2020).

BLEU, ROUGE and METEOR only credit exact words, or stems and synonyms from a fixed table. "It is freezing today" and "the weather is cold today" share two words and get a low BLEU, although they say the same thing. BERTScore replaces string equality with similarity in an encoder's representation space, where "freezing" in that context lies close to "cold".

It sits in the same place in a pipeline as the overlap metrics: a fixed test set with human references, scored after every model, prompt or decoding change. It needs a forward pass of an encoder over every candidate and reference, which is cheap on a GPU and slow on a CPU for large sets.

Greedy matching

RBERT=1∣x∣∑xi∈xmax⁡x^j∈x^xi⊤x^j,PBERT=1∣x^∣∑x^j∈x^max⁡xi∈xxi⊤x^j,FBERT=2PRP+RR_{\text{BERT}}=\frac{1}{|x|}\sum_{x_i\in x}\max_{\hat x_j\in\hat x}x_i^{\top}\hat x_j,\qquad P_{\text{BERT}}=\frac{1}{|\hat x|}\sum_{\hat x_j\in\hat x}\max_{x_i\in x}x_i^{\top}\hat x_j,\qquad F_{\text{BERT}}=\frac{2PR}{P+R}
x, x^x,\ \hat x
reference and candidate, as sequences of L2-normalised contextual token vectors from one encoder layer
xi⊤x^jx_i^{\top}\hat x_j
cosine similarity, because the vectors have unit length
RR
recall: how well each reference token is covered by some candidate token
PP
precision: how well each candidate token is supported by some reference token

Matching is greedy and many-to-one: two candidate tokens may both pick the same reference token. The paper reports that this works about as well as an optimal one-to-one matching. Embeddings come from an intermediate layer, not the last: the package picks a layer per model by correlation with human judgments on WMT16, for example layer 17 of 24 for roberta-large, its English default.

BERTScore: greedy cosine matching of contextual token embeddingsreference xtheweatheriscoldtodaycandidate x̂itisfreezingtodayencoder layer ℓ(roberta-large: layer 17 default)1 vector / tokendrop [CLS],[SEP]L2-normalisecosine xᵢ·x̂ⱼitisfreezingtodaytheweatheriscoldtoday.912.752.720.688.939.876.829.732.868.991.855.711.795.874.991.853.676.711.841.999max 0.912max 0.939max 0.991max 0.991max 0.999.939.991.991.999col maxsolid = best match per ref token · dashed = per cand tokenrecall Rmean of row maxima= 0.966precision Pmean of col maxima= 0.980F12PR / (P + R)= 0.973rescaled F(F − b) / (1 − b)b = 0.855 → 0.814R = (.912 + .939 + .991 + .991 + .999) / 5 · P = (.939 + .991 + .991 + .999) / 4IDF option: weight each token's max by idf(w) and divide by Σ idf · numbers from a toy encoder
fig.Both sentences go through the same encoder; each token becomes one unit vector, and special tokens are dropped. The 5 × 4 cosine matrix (toy encoder) gives each reference token its best candidate match, mean 0.966 = recall, and each candidate token its best reference match, mean 0.980 = precision. F1 is 0.973. Subtracting a baseline of 0.855 and dividing by 1 − 0.855 gives 0.814. Note that 'the' and 'weather' both match 'it'.
RBERTidf=∑xi∈xidf⁡(xi)max⁡x^jxi⊤x^j∑xi∈xidf⁡(xi),s^=s−b1−bR_{\text{BERT}}^{\text{idf}}=\frac{\sum_{x_i\in x}\operatorname{idf}(x_i)\max_{\hat x_j}x_i^{\top}\hat x_j}{\sum_{x_i\in x}\operatorname{idf}(x_i)},\qquad \hat s=\frac{s-b}{1-b}
idf⁡(w)\operatorname{idf}(w)
inverse document frequency over the test-set references; the package uses log⁡M+1df(w)+1\log\frac{M+1}{\text{df}(w)+1} for MM references
bb
baseline: the average score of random, unrelated sentence pairs (computed once per model and layer on Common Crawl text)
s^\hat s
rescaled score; a linear map, so rankings and correlations do not change

IDF weighting stops frequent tokens such as "the" and punctuation, which almost always find a good match, from dominating the average. The paper found it helps only slightly and not on every dataset, so it is off by default. Rescaling matters for readability. Encoder embeddings are anisotropic: all vectors share a common direction, so even unrelated sentences score around 0.8 or more. Raw scores then crowd into a narrow band near the top. After rescaling, an unrelated pair is near 0 (it can go below) and the spread between systems is visible.

The bert_score package

Default models
roberta-large (layer 17) for lang="en"; bert-base-multilingual-cased for other languages. The README recommends microsoft/deberta-xlarge-mnli for the best correlation
Hash
the package prints a hash string with the model, layer, IDF setting, rescaling and version; scores are only comparable when the hash matches
Multiple references
score() accepts several references per candidate and keeps the maximum over them
Length
inputs longer than the encoder limit (512 tokens for BERT-style models, special tokens included) are truncated, so long documents need chunking
Paper evaluation
WMT16–18 machine translation and COCO 2015 image captioning, where it matched human judgments better than n-gram metrics; plus robustness checks on the PAWS adversarial paraphrase set

Try it

cosine(xᵢ, x̂ⱼ) between contextual token vectors · β = 0.6ref: the weather is cold today · cand: it is freezing todaycandidate →itisfreezingtodaytheweatheriscoldtodayreference ↓.912.752.720.688.939.876.829.732.868.991.855.711.795.874.991.853.676.711.841.9990.9120.9390.9910.9910.999.939.991.991.999col maxrescaled (s − b)/(1 − b)R ref → cand0.767raw 0.966P cand → ref0.862raw 0.980F 2PR/(P+R)0.814raw 0.973baseline b (F) = 0.855mean over 56 randompairs of toy sentencessolid = row max (R)dashed = col max (P)toy encoder: vᵢ = eᵢ + β·(eᵢ₋₁ + eᵢ₊₁)/2, normalised; all words share a common direction (anisotropy)
pair
weightsscores
fig.A toy encoder: hand-set word vectors with a shared direction, mixed with their neighbours by weight β. Default (paraphrase, β = 0.6, rescaled): raw R 0.966, P 0.980, F 0.973; rescaled with the toy baselines (b ≈ 0.855) they become 0.767, 0.862 and 0.814. 'antonym' (cold → hot) still gets raw F 0.950, rescaled 0.654: one changed token barely moves a token-averaged score. 'unrelated' gets raw 0.807, which looks high until rescaling puts it at −0.332. 'swapped roles' (dog chased cat vs cat chased dog) scores 1.000 at β = 0, since without context it is the same bag of vectors; at β = 0.6 it scores 0.984, and 0.971 at β = 1.
StepValuesResult
row maxima (ref → cand)the .912, weather .939, is .991, cold .991, today .999R = 4.832 / 5 = 0.966
column maxima (cand → ref)it .939, is .991, freezing .991, today .999P = 3.920 / 4 = 0.980
F12 · 0.980 · 0.966 / (0.980 + 0.966)F = 0.973
baselinemean F of the 56 ordered pairs of 8 unrelated toy sentencesb = 0.855
rescaled(0.973 − 0.855) / (1 − 0.855)0.814
Worked example: the diagram's cosine matrix (reference rows, candidate columns), no IDF. Checked in Python

On the same pair, sentence BLEU with exp smoothing is 0.148 (no bigram matches at all) and ROUGE-1 F1 is 2⋅24⋅25/(24+25)=0.4442\cdot\frac{2}{4}\cdot\frac{2}{5}/(\frac{2}{4}+\frac{2}{5})=0.444 (only "is" and "today" match). Recall here is limited by "the": it has no counterpart in the candidate and its best match is "it" at 0.912.

python
1import numpy as np
2
3def bertscore_from_embeddings(ref_emb, cand_emb, ref_idf=None, cand_idf=None):
4 """ref_emb: (m, d), cand_emb: (n, d) contextual token vectors, special tokens removed."""
5 x = ref_emb / np.linalg.norm(ref_emb, axis=1, keepdims=True)
6 y = cand_emb / np.linalg.norm(cand_emb, axis=1, keepdims=True)
7 sim = x @ y.T # (m, n) cosine matrix
8 wr = np.ones(len(x)) if ref_idf is None else np.asarray(ref_idf)
9 wc = np.ones(len(y)) if cand_idf is None else np.asarray(cand_idf)
10 R = (wr * sim.max(axis=1)).sum() / wr.sum() # each reference token's best match
11 P = (wc * sim.max(axis=0)).sum() / wc.sum() # each candidate token's best match
12 return P, R, 2 * P * R / (P + R)
13
14def rescale(s, b): # baseline b from random sentence pairs
15 return (s - b) / (1 - b)
16
17# with the real package (needs torch + transformers; not run here):
18# from bert_score import score
19# P, R, F = score(cands, refs, lang="en", rescale_with_baseline=True)
20rng = np.random.default_rng(0)
21ref = rng.normal(size=(5, 64)) + 1.0 # +1.0: a shared direction, like real encoders
22cand = ref[1:] + 0.3 * rng.normal(size=(4, 64)) # candidate = ref tokens 2-5, perturbed
23P, R, F = bertscore_from_embeddings(ref, cand)
24print(round(P, 3), round(R, 3), round(F, 3)) # 0.979 0.88 0.927
BERTScore from token embeddings in numpy (run here on random vectors). The real package call is in the comment; it needs torch and transformers and was not run

Reading P, R and F

The three numbers diagnose different failures. Low recall means reference content is missing from the output, which matters for summaries and answers that must cover given points. Low precision means the output contains material the reference does not support: padding, digressions or invented details. Most reports quote F1; when the reference is a checklist of points that must appear, recall alone is the more direct signal.

Where it's used

Machine translation and paraphrase

segment- and system-level scoring where wording legitimately varies

Captioning and summarisation

a semantic companion to BLEU, CIDEr or ROUGE on the same references

Prompt and model regression tests

track mean rescaled F on a fixed set after each change

Use when

  • +Credits paraphrases and synonyms in context, with no hand-made synonym table
  • +Token-level P and R show whether output is missing content or adding unsupported content
  • +Correlates better with human judgments than n-gram metrics on MT and captioning
  • +Works for any language the encoder covers

Avoid when

  • −Needs an encoder forward pass per text; GPU helps for large sets
  • −Insensitive to negation, numbers and small factual edits
  • −Raw scores sit in a narrow high band; need baseline rescaling to read
  • −Values change with the model, layer and package version

Check yourself

A candidate copies the reference and then adds a long unsupported paragraph. Which BERTScore component drops most?

Check yourself

Why does rescaling with a baseline not change which of two systems wins?

Remember

  • ✓R = mean over reference tokens of the best cosine with a candidate token; P is the reverse; F = 2PR/(P + R).
  • ✓Vectors come from an intermediate encoder layer (roberta-large layer 17 by default for English), L2-normalised.
  • ✓Optional IDF weights from the references down-weight frequent tokens; baseline rescaling (s − b)/(1 − b) spreads scores without changing rankings.
  • ✓It credits paraphrase that BLEU and ROUGE miss, but negation, numbers and swapped roles barely change it.
  • ✓Pin the model, layer and version (the hash) and compare scores only under the same setup.
← back to Generative AI roadmap