# Semantic similarity

> Score how close two texts are in meaning: a cross-encoder reads both at once, a bi-encoder compares embeddings by cosine; evaluated by Spearman ρ against 0–5 human ratings (STS-B) or F1 on paraphrase pairs.

Source: https://www.mlfoundry.in/learn/semantic-similarity  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 7 min read · updated 2026-10-15

**Semantic textual similarity (STS):** is the task of predicting a graded score for how close two sentences are in meaning, from 0 (unrelated) to 5 (equivalent), as in the SemEval STS tasks of 2012–2017 and the STS Benchmark (Cer et al. 2017). **Paraphrase detection** is its binary cousin: does pair $(a,b)$ say the same thing?

## The task

Input: a sentence pair. Output: a real score, or a yes/no label. “A man is playing a guitar” and “A man plays the guitar” rate near 5. “The dog bites the man” and “The man bites the dog” use the same words and rate low.

Meaning survives large changes in wording (synonyms, voice, word order) but not small ones (negation, swapped roles). Counting shared words therefore fails both ways. Pre-neural systems combined word overlap, WordNet similarity and alignment features in a regressor, and **Word embeddings** later made soft word matching possible.

## Cross-encoders and bi-encoders

A **cross-encoder** feeds `[CLS] a [SEP] b [SEP]` through one transformer, so every token of $a$ attends to every token of $b$, and a linear head on `[CLS]` regresses the score. This is how **BERT** is fine-tuned on STS-B. It is the most accurate option, but it needs one full forward pass per pair.

A **bi-encoder** encodes each sentence on its own into a vector, $u=f(a)$ and $v=f(b)$, and compares them with cosine similarity. Sentence-BERT (Reimers and Gurevych 2019) trains BERT this way in a Siamese setup with shared weights. How such encoders are trained is covered in **Sentence embeddings** and **Contrastive language learning**.

> Figure: Left: the cross-encoder reads both sentences jointly and costs one pass per pair. Right: the bi-encoder (Sentence-BERT) encodes each sentence once, pools to u and v, and compares by cosine. Below: STS-B, paraphrase detection and semantic search all consume the same score.

$$
\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert\,\lVert v\rVert}\in[-1,1],\qquad \mathcal{L}_{\text{reg}}=\big(\cos(u,v)-y\big)^2
$$

- $u,\;v$: pooled sentence embeddings (often the mean of the token vectors)
- $y$: gold similarity rescaled to $[0,1]$, e.g. STS score / 5
- $\mathcal{L}_{\text{reg}}$: Sentence-BERT's regression objective; a cross-encoder instead applies MSE to the [CLS] head's output

Cost decides which to use. Finding the most similar pair among 10,000 sentences takes $n(n-1)/2=49{,}995{,}000$ cross-encoder passes, about 65 hours with BERT in the Sentence-BERT paper, against about 5 seconds with a bi-encoder. The usual pattern is to retrieve candidates with a bi-encoder and re-score the top few with a cross-encoder.

## Evaluation

STS systems are scored by how well their scores correlate with the human ones. Spearman's $\rho$ is Pearson's correlation computed on ranks. It only asks whether the model *orders* the pairs correctly, so any monotone rescaling of the cosine leaves it unchanged. Sentence-BERT and most embedding papers report it.

$$
\rho=1-\frac{6\sum_i d_i^2}{n(n^2-1)}
$$

- $d_i$: difference between the rank of pair i by gold score and by model score
- $n$: number of pairs (the formula assumes no ties; with ties, use Pearson on average ranks)

**Datasets and reference numbers**

- **STS-B:** Cer, Diab, Agirre, Lopez-Gazpio, Specia 2017: 8,628 pairs from news, captions and forums (5,749 train / 1,500 dev / 1,379 test), gold 0–5. Part of GLUE.
- **MRPC:** Dolan and Brockett 2005: 5,801 news sentence pairs labelled paraphrase or not, about two-thirds positive. Reported as F1 and accuracy.
- **QQP:** Quora Question Pairs: over 400,000 question pairs labelled duplicate or not. Reported as F1 and accuracy in GLUE.
- **PAWS:** Zhang, Baldridge, He 2019: pairs made by swapping words, so lexical overlap is high whatever the label. Bag-of-words models fail on it.
- **Cross-encoder:** BERT-large fine-tuned on STS-B: 86.5 Spearman on the GLUE test set (Devlin et al. 2019).
- **Bi-encoder, no STS training:** STS-B test Spearman ×100 (Reimers and Gurevych 2019): averaged GloVe 58.02, averaged BERT token vectors 46.35, SBERT-NLI-base 77.03.

The last row is the surprise. Averaging raw BERT outputs is worse than averaging GloVe vectors. Raw contextual embeddings are **anisotropic**: they share a dominant direction, so every pair has a high cosine (Ethayarajh 2019). Training for similarity, or removing the shared component, spreads them out.

> Figure: Ten STS-style pairs scored by a toy bi-encoder: 8-d word vectors, mean-pooled, then cosine. By default Pearson r = 0.616 and Spearman ρ = 0.503, and at threshold 0.80 the paraphrase decisions have 4 false positives (accuracy 0.60, F1 0.667). Dropping stop-words lifts ρ to 0.709 and F1 to 0.889. A shared direction of size α = 2 pushes every cosine to 0.866 or more, so ρ falls to 0.297 and accuracy to 0.40. Subtract mean cancels the shared direction at any α (ρ = 0.491). “Dog bites man” vs its reverse always scores 1.000.

## Worked example: Spearman's ρ by hand

The figure's default cosines. Ranks are ascending (1 = least similar); d = gold rank − model rank.

| pair | gold | cos | gold rank | cos rank | d² |
| --- | --- | --- | --- | --- | --- |
| plane ↔ airplane | 5.0 | 0.932 | 10 | 7 | 9 |
| playing ↔ plays guitar | 4.8 | 0.873 | 9 | 5 | 16 |
| slicing ↔ cutting onion | 4.6 | 0.994 | 8 | 9 | 1 |
| stocks fell ↔ market dropped | 4.2 | 0.937 | 7 | 8 | 1 |
| cat sits ↔ cat sleeping | 3.4 | 0.775 | 6 | 2 | 16 |
| reading ↔ writing a book | 2.6 | 0.910 | 5 | 6 | 1 |
| dog bites man ↔ reversed | 1.8 | 1.000 | 4 | 10 | 36 |
| boy/horse ↔ girl/bike | 1.6 | 0.830 | 3 | 4 | 1 |
| woman dancing ↔ man talking | 0.8 | 0.803 | 2 | 3 | 1 |
| child pasta ↔ sun sea | 0.2 | 0.322 | 1 | 1 | 0 |

$\sum d^2=82$, so $\rho=1-\frac{6\cdot82}{10\cdot99}=1-0.497=0.503$. The word-order pair alone contributes 36 of the 82. Mean pooling cannot see order, so this pair gets the top cosine while its gold rank is 4.

```python
from scipy.stats import spearmanr
from sentence_transformers import CrossEncoder, SentenceTransformer, util

a = ["a man is playing a guitar", "the dog bites the man"]
b = ["a man plays the guitar", "the man bites the dog"]

bi = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
u = bi.encode(a, normalize_embeddings=True)       # one pass per sentence
v = bi.encode(b, normalize_embeddings=True)
print(util.cos_sim(u, v).diagonal())               # cosine per pair

ce = CrossEncoder("cross-encoder/stsb-roberta-base")
print(ce.predict(list(zip(a, b))))                 # one pass per pair

gold = [5.0, 4.8, 4.6, 4.2, 3.4, 2.6, 1.8, 1.6, 0.8, 0.2]
cos = [0.932, 0.873, 0.994, 0.937, 0.775, 0.910, 1.000, 0.830, 0.803, 0.322]
print(round(spearmanr(gold, cos).correlation, 3))  # 0.503
```

Bi-encoder and cross-encoder scoring with sentence-transformers, and Spearman's ρ on the toy numbers (the model calls were not run here; ρ = 0.503 was checked in plain Python)

> **Thresholds do not transfer:** Spearman ignores the scale of the scores; a paraphrase threshold depends on it. A cosine of 0.8 can mean “near-duplicate” for one encoder and “barely related” for another, and it shifts if the embedding space is anisotropic. Tune the threshold on labelled pairs from your own data, for each model.

> **Similar is not the same as entailed:** STS is symmetric and graded. Entailment (NLI) is directional: “a man plays the guitar” entails “a person plays an instrument” but not the reverse. Picking the right task for the application matters as much as the model.

**Strengths**

- Bi-encoder: embed the corpus once, then search millions of items with nearest-neighbour indexes
- Cross-encoder: full token-level interaction, the most accurate score per pair
- Spearman ρ is robust to monotone rescaling, so models with different score ranges compare fairly

**Limitations**

- Bi-encoder: one vector per sentence loses detail such as word order, negation and who did what
- Cross-encoder: $O(n^2)$ passes to compare $n$ sentences; no reusable vectors
- Gold STS ratings are noisy and domain-specific; scores on STS-B do not guarantee good retrieval

**Question:** You must find near-duplicates among 1 million support tickets. Which design fits?

- A. Cross-encoder on every pair
- B. Bi-encoder embeddings plus nearest-neighbour search, then a cross-encoder on the top candidates
- C. Spearman correlation between tickets
- D. BLEU between every pair

**Answer:** B. Bi-encoder embeddings plus nearest-neighbour search, then a cross-encoder on the top candidates — All pairs is about $5\times10^{11}$ cross-encoder passes. A bi-encoder needs $10^6$ passes, nearest-neighbour search finds candidates cheaply, and the cross-encoder only re-scores a short list.

**Question:** Model A's cosines are exactly Model B's cosines squared (all positive). How do their STS-B Spearman scores compare?

- A. A is higher
- B. B is higher
- C. They are identical
- D. It depends on the threshold

**Answer:** C. They are identical — Squaring is monotone on positive numbers, so every pair keeps its rank and $\rho$ is unchanged. Pearson $r$ would change, and so would any fixed paraphrase threshold.

**Key points**

- STS: graded 0–5 similarity of a sentence pair; paraphrase detection is the binary version (MRPC, QQP).
- Cross-encoder: one joint pass per pair, most accurate. Bi-encoder: $\cos(f(a),f(b))$, one pass per sentence, searchable.
- Evaluate with Spearman $\rho=1-6\sum d_i^2/(n(n^2-1))$: rank agreement with human scores.
- Mean pooling ignores word order; raw contextual embeddings are anisotropic and need similarity training.
- Retrieve with a bi-encoder, re-rank with a cross-encoder; tune thresholds per model.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
