Semantic similarity

Score how close two texts are in meaning: a cross-encoder reads both at once, a bi-encoder compares embeddings by cosine; evaluated by Spearman ρ against 0–5 human ratings (STS-B) or F1 on paraphrase pairs.

7 min read577 wordscore0 viewsupdated 2026-10-15

Definition

Semantic textual similarity (STS) is the task of predicting a graded score for how close two sentences are in meaning, from 0 (unrelated) to 5 (equivalent), as in the SemEval STS tasks of 2012–2017 and the STS Benchmark (Cer et al. 2017). Paraphrase detection is its binary cousin: does pair (a,b)(a,b) say the same thing?

The task

Input: a sentence pair. Output: a real score, or a yes/no label. “A man is playing a guitar” and “A man plays the guitar” rate near 5. “The dog bites the man” and “The man bites the dog” use the same words and rate low.

Meaning survives large changes in wording (synonyms, voice, word order) but not small ones (negation, swapped roles). Counting shared words therefore fails both ways. Pre-neural systems combined word overlap, WordNet similarity and alignment features in a regressor, and Word embeddings later made soft word matching possible.

Cross-encoders and bi-encoders

A cross-encoder feeds [CLS] a [SEP] b [SEP] through one transformer, so every token of aa attends to every token of bb, and a linear head on [CLS] regresses the score. This is how BERT is fine-tuned on STS-B. It is the most accurate option, but it needs one full forward pass per pair.

A bi-encoder encodes each sentence on its own into a vector, u=f(a)u=f(a) and v=f(b)v=f(b), and compares them with cosine similarity. Sentence-BERT (Reimers and Gurevych 2019) trains BERT this way in a Siamese setup with shared weights. How such encoders are trained is covered in Sentence embeddings and Contrastive language learning.

cross-encoder: pair in, score out[CLS] a man is playing a guitar[SEP] a man plays the guitar [SEP]BERT, one pass over both sentencesevery token of A attends to B[CLS] → linear → score on 0–5trained with MSE against goldn sentences → n(n−1)/2 passes10,000 sentences: 49,995,000 pairs, ≈ 65 hmost accurate; used to re-rank a shortlistbi-encoder (Sentence-BERT, 2019)a man is playing…a man plays the…BERTshared weightsBERTshared weightsmean pool → umean pool → vcos(u,v) = u·v / (‖u‖‖v‖)n passes; vectors cached and indexed10,000 sentences ≈ 5 s, then cheap dot productstasks that consume the scoreSTS-B (2017)8,628 pairsgold 0 (unrelated)to 5 (equivalent)metric: Spearman ρparaphrase detectionMRPC (news), QQP(Quora questions)label 0/1metric: F1, accuracysemantic searchembed corpus once,nearest neighboursof the query vectormetric: recall@kcommon pattern: bi-encoder retrieves the top-k, cross-encoder re-scores them
fig.Left: the cross-encoder reads both sentences jointly and costs one pass per pair. Right: the bi-encoder (Sentence-BERT) encodes each sentence once, pools to u and v, and compares by cosine. Below: STS-B, paraphrase detection and semantic search all consume the same score.
cos⁡(u,v)=u⋅v∥u∥ ∥v∥∈[−1,1],Lreg=(cos⁡(u,v)−y)2\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert\,\lVert v\rVert}\in[-1,1],\qquad \mathcal{L}_{\text{reg}}=\big(\cos(u,v)-y\big)^2
u,  vu,\;v
pooled sentence embeddings (often the mean of the token vectors)
yy
gold similarity rescaled to [0,1][0,1], e.g. STS score / 5
Lreg\mathcal{L}_{\text{reg}}
Sentence-BERT's regression objective; a cross-encoder instead applies MSE to the [CLS] head's output

Cost decides which to use. Finding the most similar pair among 10,000 sentences takes n(n−1)/2=49,995,000n(n-1)/2=49{,}995{,}000 cross-encoder passes, about 65 hours with BERT in the Sentence-BERT paper, against about 5 seconds with a bi-encoder. The usual pattern is to retrieve candidates with a bi-encoder and re-score the top few with a cross-encoder.

Evaluation

STS systems are scored by how well their scores correlate with the human ones. Spearman's ρ\rho is Pearson's correlation computed on ranks. It only asks whether the model orders the pairs correctly, so any monotone rescaling of the cosine leaves it unchanged. Sentence-BERT and most embedding papers report it.

ρ=1−6∑idi2n(n2−1)\rho=1-\frac{6\sum_i d_i^2}{n(n^2-1)}
did_i
difference between the rank of pair i by gold score and by model score
nn
number of pairs (the formula assumes no ties; with ties, use Pearson on average ranks)

Datasets and reference numbers

STS-B
Cer, Diab, Agirre, Lopez-Gazpio, Specia 2017: 8,628 pairs from news, captions and forums (5,749 train / 1,500 dev / 1,379 test), gold 0–5. Part of GLUE.
MRPC
Dolan and Brockett 2005: 5,801 news sentence pairs labelled paraphrase or not, about two-thirds positive. Reported as F1 and accuracy.
QQP
Quora Question Pairs: over 400,000 question pairs labelled duplicate or not. Reported as F1 and accuracy in GLUE.
PAWS
Zhang, Baldridge, He 2019: pairs made by swapping words, so lexical overlap is high whatever the label. Bag-of-words models fail on it.
Cross-encoder
BERT-large fine-tuned on STS-B: 86.5 Spearman on the GLUE test set (Devlin et al. 2019).
Bi-encoder, no STS training
STS-B test Spearman ×100 (Reimers and Gurevych 2019): averaged GloVe 58.02, averaged BERT token vectors 46.35, SBERT-NLI-base 77.03.

The last row is the surprise. Averaging raw BERT outputs is worse than averaging GloVe vectors. Raw contextual embeddings are anisotropic: they share a dominant direction, so every pair has a high cosine (Ethayarajh 2019). Training for similarity, or removing the shared component, spreads them out.

mean-pooled word vectors → cosinePearson r = 0.616 · Spearman ρ = 0.503 · cosines span 0.32…1.00012345-0.400.51threshold 0.80gold (0–5) · right of green line = paraphrasecosinepairgoldcosplane ↔ airplane5.00.932✓playing ↔ plays guitar4.80.873✓slicing ↔ cutting onion4.60.994✓stocks fell ↔ market dropped4.20.937✓cat sits ↔ cat sleeping3.40.775✓reading ↔ writing a book2.60.910FPdog bites man ↔ reversed1.81.000FPboy/horse ↔ girl/bike1.60.830FPwoman dancing ↔ man talking0.80.803FPchild pasta ↔ sun sea0.20.322✓paraphrase if cos ≥ 0.80: TP 4 · FP 4 · FN 0 · TN 2 → accuracy 0.60, F1 0.667word order is invisible to mean pooling: “dog bites man” vs its reverse scores 1.000filled dot = predicted paraphrase · gold scores are illustrative, in the STS-B 0–5 style
pooling
fig.Ten STS-style pairs scored by a toy bi-encoder: 8-d word vectors, mean-pooled, then cosine. By default Pearson r = 0.616 and Spearman ρ = 0.503, and at threshold 0.80 the paraphrase decisions have 4 false positives (accuracy 0.60, F1 0.667). Dropping stop-words lifts ρ to 0.709 and F1 to 0.889. A shared direction of size α = 2 pushes every cosine to 0.866 or more, so ρ falls to 0.297 and accuracy to 0.40. Subtract mean cancels the shared direction at any α (ρ = 0.491). “Dog bites man” vs its reverse always scores 1.000.

Worked example: Spearman's ρ by hand

pairgoldcosgold rankcos rankd²
plane ↔ airplane5.00.9321079
playing ↔ plays guitar4.80.8739516
slicing ↔ cutting onion4.60.994891
stocks fell ↔ market dropped4.20.937781
cat sits ↔ cat sleeping3.40.7756216
reading ↔ writing a book2.60.910561
dog bites man ↔ reversed1.81.00041036
boy/horse ↔ girl/bike1.60.830341
woman dancing ↔ man talking0.80.803231
child pasta ↔ sun sea0.20.322110
The figure's default cosines. Ranks are ascending (1 = least similar); d = gold rank − model rank.

∑d2=82\sum d^2=82, so ρ=1−6⋅8210⋅99=1−0.497=0.503\rho=1-\frac{6\cdot82}{10\cdot99}=1-0.497=0.503. The word-order pair alone contributes 36 of the 82. Mean pooling cannot see order, so this pair gets the top cosine while its gold rank is 4.

python
1from scipy.stats import spearmanr
2from sentence_transformers import CrossEncoder, SentenceTransformer, util
3
4a = ["a man is playing a guitar", "the dog bites the man"]
5b = ["a man plays the guitar", "the man bites the dog"]
6
7bi = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
8u = bi.encode(a, normalize_embeddings=True) # one pass per sentence
9v = bi.encode(b, normalize_embeddings=True)
10print(util.cos_sim(u, v).diagonal()) # cosine per pair
11
12ce = CrossEncoder("cross-encoder/stsb-roberta-base")
13print(ce.predict(list(zip(a, b)))) # one pass per pair
14
15gold = [5.0, 4.8, 4.6, 4.2, 3.4, 2.6, 1.8, 1.6, 0.8, 0.2]
16cos = [0.932, 0.873, 0.994, 0.937, 0.775, 0.910, 1.000, 0.830, 0.803, 0.322]
17print(round(spearmanr(gold, cos).correlation, 3)) # 0.503
Bi-encoder and cross-encoder scoring with sentence-transformers, and Spearman's ρ on the toy numbers (the model calls were not run here; ρ = 0.503 was checked in plain Python)

Use when

  • +Bi-encoder: embed the corpus once, then search millions of items with nearest-neighbour indexes
  • +Cross-encoder: full token-level interaction, the most accurate score per pair
  • +Spearman ρ is robust to monotone rescaling, so models with different score ranges compare fairly

Avoid when

  • −Bi-encoder: one vector per sentence loses detail such as word order, negation and who did what
  • −Cross-encoder: O(n2)O(n^2) passes to compare nn sentences; no reusable vectors
  • −Gold STS ratings are noisy and domain-specific; scores on STS-B do not guarantee good retrieval

Check yourself

You must find near-duplicates among 1 million support tickets. Which design fits?

Check yourself

Model A's cosines are exactly Model B's cosines squared (all positive). How do their STS-B Spearman scores compare?

Remember

  • ✓STS: graded 0–5 similarity of a sentence pair; paraphrase detection is the binary version (MRPC, QQP).
  • ✓Cross-encoder: one joint pass per pair, most accurate. Bi-encoder: cos⁡(f(a),f(b))\cos(f(a),f(b)), one pass per sentence, searchable.
  • ✓Evaluate with Spearman ρ=1−6∑di2/(n(n2−1))\rho=1-6\sum d_i^2/(n(n^2-1)): rank agreement with human scores.
  • ✓Mean pooling ignores word order; raw contextual embeddings are anisotropic and need similarity training.
  • ✓Retrieve with a bi-encoder, re-rank with a cross-encoder; tune thresholds per model.
← back to Deep Learning roadmap