Definition
BERTScore A reference-based text-generation metric that compares contextual token embeddings instead of surface strings. Each token is matched greedily to its most similar token in the other text; the averages give precision, recall and F1. Introduced by Zhang, Kishore, Wu, Weinberger and Artzi, "BERTScore: Evaluating Text Generation with BERT" (ICLR 2020).
BLEU, ROUGE and METEOR only credit exact words, or stems and synonyms from a fixed table. "It is freezing today" and "the weather is cold today" share two words and get a low BLEU, although they say the same thing. BERTScore replaces string equality with similarity in an encoder's representation space, where "freezing" in that context lies close to "cold".
It sits in the same place in a pipeline as the overlap metrics: a fixed test set with human references, scored after every model, prompt or decoding change. It needs a forward pass of an encoder over every candidate and reference, which is cheap on a GPU and slow on a CPU for large sets.
Greedy matching
- reference and candidate, as sequences of L2-normalised contextual token vectors from one encoder layer
- cosine similarity, because the vectors have unit length
- recall: how well each reference token is covered by some candidate token
- precision: how well each candidate token is supported by some reference token
Matching is greedy and many-to-one: two candidate tokens may both pick the same reference token. The paper reports that this works about as well as an optimal one-to-one matching. Embeddings come from an intermediate layer, not the last: the package picks a layer per model by correlation with human judgments on WMT16, for example layer 17 of 24 for roberta-large, its English default.
- inverse document frequency over the test-set references; the package uses for references
- baseline: the average score of random, unrelated sentence pairs (computed once per model and layer on Common Crawl text)
- rescaled score; a linear map, so rankings and correlations do not change
IDF weighting stops frequent tokens such as "the" and punctuation, which almost always find a good match, from dominating the average. The paper found it helps only slightly and not on every dataset, so it is off by default. Rescaling matters for readability. Encoder embeddings are anisotropic: all vectors share a common direction, so even unrelated sentences score around 0.8 or more. Raw scores then crowd into a narrow band near the top. After rescaling, an unrelated pair is near 0 (it can go below) and the spread between systems is visible.
The bert_score package
- Default models
- roberta-large (layer 17) for lang="en"; bert-base-multilingual-cased for other languages. The README recommends microsoft/deberta-xlarge-mnli for the best correlation
- Hash
- the package prints a hash string with the model, layer, IDF setting, rescaling and version; scores are only comparable when the hash matches
- Multiple references
- score() accepts several references per candidate and keeps the maximum over them
- Length
- inputs longer than the encoder limit (512 tokens for BERT-style models, special tokens included) are truncated, so long documents need chunking
- Paper evaluation
- WMT16–18 machine translation and COCO 2015 image captioning, where it matched human judgments better than n-gram metrics; plus robustness checks on the PAWS adversarial paraphrase set
Try it
| Step | Values | Result |
|---|---|---|
| row maxima (ref → cand) | the .912, weather .939, is .991, cold .991, today .999 | R = 4.832 / 5 = 0.966 |
| column maxima (cand → ref) | it .939, is .991, freezing .991, today .999 | P = 3.920 / 4 = 0.980 |
| F1 | 2 · 0.980 · 0.966 / (0.980 + 0.966) | F = 0.973 |
| baseline | mean F of the 56 ordered pairs of 8 unrelated toy sentences | b = 0.855 |
| rescaled | (0.973 − 0.855) / (1 − 0.855) | 0.814 |
On the same pair, sentence BLEU with exp smoothing is 0.148 (no bigram matches at all) and ROUGE-1 F1 is (only "is" and "today" match). Recall here is limited by "the": it has no counterpart in the candidate and its best match is "it" at 0.912.
1import numpy as np23def bertscore_from_embeddings(ref_emb, cand_emb, ref_idf=None, cand_idf=None):4 """ref_emb: (m, d), cand_emb: (n, d) contextual token vectors, special tokens removed."""5 x = ref_emb / np.linalg.norm(ref_emb, axis=1, keepdims=True)6 y = cand_emb / np.linalg.norm(cand_emb, axis=1, keepdims=True)7 sim = x @ y.T # (m, n) cosine matrix8 wr = np.ones(len(x)) if ref_idf is None else np.asarray(ref_idf)9 wc = np.ones(len(y)) if cand_idf is None else np.asarray(cand_idf)10 R = (wr * sim.max(axis=1)).sum() / wr.sum() # each reference token's best match11 P = (wc * sim.max(axis=0)).sum() / wc.sum() # each candidate token's best match12 return P, R, 2 * P * R / (P + R)1314def rescale(s, b): # baseline b from random sentence pairs15 return (s - b) / (1 - b)1617# with the real package (needs torch + transformers; not run here):18# from bert_score import score19# P, R, F = score(cands, refs, lang="en", rescale_with_baseline=True)20rng = np.random.default_rng(0)21ref = rng.normal(size=(5, 64)) + 1.0 # +1.0: a shared direction, like real encoders22cand = ref[1:] + 0.3 * rng.normal(size=(4, 64)) # candidate = ref tokens 2-5, perturbed23P, R, F = bertscore_from_embeddings(ref, cand)24print(round(P, 3), round(R, 3), round(F, 3)) # 0.979 0.88 0.927Reading P, R and F
The three numbers diagnose different failures. Low recall means reference content is missing from the output, which matters for summaries and answers that must cover given points. Low precision means the output contains material the reference does not support: padding, digressions or invented details. Most reports quote F1; when the reference is a checklist of points that must appear, recall alone is the more direct signal.
Where it's used
Machine translation and paraphrase
segment- and system-level scoring where wording legitimately varies
Captioning and summarisation
a semantic companion to BLEU, CIDEr or ROUGE on the same references
Prompt and model regression tests
track mean rescaled F on a fixed set after each change
Use when
- +Credits paraphrases and synonyms in context, with no hand-made synonym table
- +Token-level P and R show whether output is missing content or adding unsupported content
- +Correlates better with human judgments than n-gram metrics on MT and captioning
- +Works for any language the encoder covers
Avoid when
- −Needs an encoder forward pass per text; GPU helps for large sets
- −Insensitive to negation, numbers and small factual edits
- −Raw scores sit in a narrow high band; need baseline rescaling to read
- −Values change with the model, layer and package version
Check yourself
A candidate copies the reference and then adds a long unsupported paragraph. Which BERTScore component drops most?
Check yourself
Why does rescaling with a baseline not change which of two systems wins?
Remember
- ✓R = mean over reference tokens of the best cosine with a candidate token; P is the reverse; F = 2PR/(P + R).
- ✓Vectors come from an intermediate encoder layer (roberta-large layer 17 by default for English), L2-normalised.
- ✓Optional IDF weights from the references down-weight frequent tokens; baseline rescaling (s − b)/(1 − b) spreads scores without changing rankings.
- ✓It credits paraphrase that BLEU and ROUGE miss, but negation, numbers and swapped roles barely change it.
- ✓Pin the model, layer and version (the hash) and compare scores only under the same setup.