# BERTScore

> BERTScore matches every token of candidate and reference to its most similar token in the other by cosine of contextual embeddings; mean best matches give precision, recall and F1, so paraphrases score high where n-gram overlap fails.

Source: https://www.mlfoundry.in/learn/bertscore  
Section: Generative AI › Evaluation of Generative Models  
Level: core · 7 min read · updated 2026-10-21

**BERTScore:** A reference-based text-generation metric that compares **contextual token embeddings** instead of surface strings. Each token is matched greedily to its most similar token in the other text; the averages give precision, recall and F1. Introduced by Zhang, Kishore, Wu, Weinberger and Artzi, "BERTScore: Evaluating Text Generation with BERT" (ICLR 2020).

**BLEU, ROUGE and METEOR** only credit exact words, or stems and synonyms from a fixed table. "It is freezing today" and "the weather is cold today" share two words and get a low BLEU, although they say the same thing. BERTScore replaces string equality with similarity in an encoder's representation space, where "freezing" in that context lies close to "cold".

It sits in the same place in a pipeline as the overlap metrics: a fixed test set with human references, scored after every model, prompt or decoding change. It needs a forward pass of an encoder over every candidate and reference, which is cheap on a GPU and slow on a CPU for large sets.

## Greedy matching

$$
R_{\text{BERT}}=\frac{1}{|x|}\sum_{x_i\in x}\max_{\hat x_j\in\hat x}x_i^{\top}\hat x_j,\qquad P_{\text{BERT}}=\frac{1}{|\hat x|}\sum_{\hat x_j\in\hat x}\max_{x_i\in x}x_i^{\top}\hat x_j,\qquad F_{\text{BERT}}=\frac{2PR}{P+R}
$$

- $x,\ \hat x$: reference and candidate, as sequences of L2-normalised contextual token vectors from one encoder layer
- $x_i^{\top}\hat x_j$: cosine similarity, because the vectors have unit length
- $R$: recall: how well each reference token is covered by some candidate token
- $P$: precision: how well each candidate token is supported by some reference token

Matching is greedy and many-to-one: two candidate tokens may both pick the same reference token. The paper reports that this works about as well as an optimal one-to-one matching. Embeddings come from an intermediate layer, not the last: the package picks a layer per model by correlation with human judgments on WMT16, for example layer 17 of 24 for `roberta-large`, its English default.

> Figure: Both sentences go through the same encoder; each token becomes one unit vector, and special tokens are dropped. The 5 × 4 cosine matrix (toy encoder) gives each reference token its best candidate match, mean 0.966 = recall, and each candidate token its best reference match, mean 0.980 = precision. F1 is 0.973. Subtracting a baseline of 0.855 and dividing by 1 − 0.855 gives 0.814. Note that 'the' and 'weather' both match 'it'.

$$
R_{\text{BERT}}^{\text{idf}}=\frac{\sum_{x_i\in x}\operatorname{idf}(x_i)\max_{\hat x_j}x_i^{\top}\hat x_j}{\sum_{x_i\in x}\operatorname{idf}(x_i)},\qquad \hat s=\frac{s-b}{1-b}
$$

- $\operatorname{idf}(w)$: inverse document frequency over the test-set references; the package uses $\log\frac{M+1}{\text{df}(w)+1}$ for $M$ references
- $b$: baseline: the average score of random, unrelated sentence pairs (computed once per model and layer on Common Crawl text)
- $\hat s$: rescaled score; a linear map, so rankings and correlations do not change

IDF weighting stops frequent tokens such as "the" and punctuation, which almost always find a good match, from dominating the average. The paper found it helps only slightly and not on every dataset, so it is off by default. Rescaling matters for readability. Encoder embeddings are anisotropic: all vectors share a common direction, so even unrelated sentences score around 0.8 or more. Raw scores then crowd into a narrow band near the top. After rescaling, an unrelated pair is near 0 (it can go below) and the spread between systems is visible.

**The bert_score package**

- **Default models:** roberta-large (layer 17) for lang="en"; bert-base-multilingual-cased for other languages. The README recommends microsoft/deberta-xlarge-mnli for the best correlation
- **Hash:** the package prints a hash string with the model, layer, IDF setting, rescaling and version; scores are only comparable when the hash matches
- **Multiple references:** score() accepts several references per candidate and keeps the maximum over them
- **Length:** inputs longer than the encoder limit (512 tokens for BERT-style models, special tokens included) are truncated, so long documents need chunking
- **Paper evaluation:** WMT16–18 machine translation and COCO 2015 image captioning, where it matched human judgments better than n-gram metrics; plus robustness checks on the PAWS adversarial paraphrase set

## Try it

> Figure: A toy encoder: hand-set word vectors with a shared direction, mixed with their neighbours by weight β. Default (paraphrase, β = 0.6, rescaled): raw R 0.966, P 0.980, F 0.973; rescaled with the toy baselines (b ≈ 0.855) they become 0.767, 0.862 and 0.814. 'antonym' (cold → hot) still gets raw F 0.950, rescaled 0.654: one changed token barely moves a token-averaged score. 'unrelated' gets raw 0.807, which looks high until rescaling puts it at −0.332. 'swapped roles' (dog chased cat vs cat chased dog) scores 1.000 at β = 0, since without context it is the same bag of vectors; at β = 0.6 it scores 0.984, and 0.971 at β = 1.

Worked example: the diagram's cosine matrix (reference rows, candidate columns), no IDF. Checked in Python

| Step | Values | Result |
| --- | --- | --- |
| row maxima (ref → cand) | the .912, weather .939, is .991, cold .991, today .999 | R = 4.832 / 5 = 0.966 |
| column maxima (cand → ref) | it .939, is .991, freezing .991, today .999 | P = 3.920 / 4 = 0.980 |
| F1 | 2 · 0.980 · 0.966 / (0.980 + 0.966) | F = 0.973 |
| baseline | mean F of the 56 ordered pairs of 8 unrelated toy sentences | b = 0.855 |
| rescaled | (0.973 − 0.855) / (1 − 0.855) | 0.814 |

On the same pair, sentence BLEU with exp smoothing is 0.148 (no bigram matches at all) and ROUGE-1 F1 is $2\cdot\frac{2}{4}\cdot\frac{2}{5}/(\frac{2}{4}+\frac{2}{5})=0.444$ (only "is" and "today" match). Recall here is limited by "the": it has no counterpart in the candidate and its best match is "it" at 0.912.

```python
import numpy as np

def bertscore_from_embeddings(ref_emb, cand_emb, ref_idf=None, cand_idf=None):
    """ref_emb: (m, d), cand_emb: (n, d) contextual token vectors, special tokens removed."""
    x = ref_emb / np.linalg.norm(ref_emb, axis=1, keepdims=True)
    y = cand_emb / np.linalg.norm(cand_emb, axis=1, keepdims=True)
    sim = x @ y.T                                    # (m, n) cosine matrix
    wr = np.ones(len(x)) if ref_idf is None else np.asarray(ref_idf)
    wc = np.ones(len(y)) if cand_idf is None else np.asarray(cand_idf)
    R = (wr * sim.max(axis=1)).sum() / wr.sum()      # each reference token's best match
    P = (wc * sim.max(axis=0)).sum() / wc.sum()      # each candidate token's best match
    return P, R, 2 * P * R / (P + R)

def rescale(s, b):                                    # baseline b from random sentence pairs
    return (s - b) / (1 - b)

# with the real package (needs torch + transformers; not run here):
#   from bert_score import score
#   P, R, F = score(cands, refs, lang="en", rescale_with_baseline=True)
rng = np.random.default_rng(0)
ref = rng.normal(size=(5, 64)) + 1.0                  # +1.0: a shared direction, like real encoders
cand = ref[1:] + 0.3 * rng.normal(size=(4, 64))      # candidate = ref tokens 2-5, perturbed
P, R, F = bertscore_from_embeddings(ref, cand)
print(round(P, 3), round(R, 3), round(F, 3))         # 0.979 0.88 0.927
```

BERTScore from token embeddings in numpy (run here on random vectors). The real package call is in the comment; it needs torch and transformers and was not run

## Reading P, R and F

The three numbers diagnose different failures. Low **recall** means reference content is missing from the output, which matters for summaries and answers that must cover given points. Low **precision** means the output contains material the reference does not support: padding, digressions or invented details. Most reports quote F1; when the reference is a checklist of points that must appear, recall alone is the more direct signal.

- **Machine translation and paraphrase:** segment- and system-level scoring where wording legitimately varies
- **Captioning and summarisation:** a semantic companion to BLEU, CIDEr or ROUGE on the same references
- **Prompt and model regression tests:** track mean rescaled F on a fixed set after each change

> **High similarity is not correctness:** BERTScore rewards shared meaning *on average over tokens*. Negation, a swapped number, a wrong entity or reversed roles change one or two tokens, and the score barely moves: in the figure, "cold" → "hot" keeps raw F at 0.950. Do not use it as a factuality metric; use **Hallucination and groundedness evaluation** or **Factuality evaluation** for that.

> **Scores are model-specific:** Raw values depend on the encoder, the layer and the tokenizer. 0.90 from `roberta-large` and 0.90 from `deberta-xlarge-mnli` mean different things. Fix the model and version in the evaluation config, report the hash, and compare systems only under the same one.

**Strengths**

- Credits paraphrases and synonyms in context, with no hand-made synonym table
- Token-level P and R show whether output is missing content or adding unsupported content
- Correlates better with human judgments than n-gram metrics on MT and captioning
- Works for any language the encoder covers

**Limitations**

- Needs an encoder forward pass per text; GPU helps for large sets
- Insensitive to negation, numbers and small factual edits
- Raw scores sit in a narrow high band; need baseline rescaling to read
- Values change with the model, layer and package version

**Question:** A candidate copies the reference and then adds a long unsupported paragraph. Which BERTScore component drops most?

- A. Recall
- B. Precision
- C. Both by the same amount
- D. Neither, since every reference token is matched

**Answer:** B. Precision — Every reference token still finds its copy, so recall stays near 1. The added candidate tokens have no good reference match, so their column maxima are low and precision falls. F1 falls with it.

**Question:** Why does rescaling with a baseline not change which of two systems wins?

- A. Because the baseline is subtracted only from the worse system
- B. Because (s − b)/(1 − b) is an increasing linear map of s with the same b for both
- C. Because rescaling switches the metric from F1 to recall
- D. It does change the winner when IDF is on

**Answer:** B. Because (s − b)/(1 − b) is an increasing linear map of s with the same b for both — For a fixed model and layer, b is one constant, so the rescaled score is an affine, increasing function of the raw score. Order and correlation with humans are unchanged; only readability improves.

**Key points**

- R = mean over reference tokens of the best cosine with a candidate token; P is the reverse; F = 2PR/(P + R).
- Vectors come from an intermediate encoder layer (roberta-large layer 17 by default for English), L2-normalised.
- Optional IDF weights from the references down-weight frequent tokens; baseline rescaling (s − b)/(1 − b) spreads scores without changing rankings.
- It credits paraphrase that BLEU and ROUGE miss, but negation, numbers and swapped roles barely change it.
- Pin the model, layer and version (the hash) and compare scores only under the same setup.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
