Definition
Semantic textual similarity (STS) is the task of predicting a graded score for how close two sentences are in meaning, from 0 (unrelated) to 5 (equivalent), as in the SemEval STS tasks of 2012–2017 and the STS Benchmark (Cer et al. 2017). Paraphrase detection is its binary cousin: does pair say the same thing?
The task
Input: a sentence pair. Output: a real score, or a yes/no label. “A man is playing a guitar” and “A man plays the guitar” rate near 5. “The dog bites the man” and “The man bites the dog” use the same words and rate low.
Meaning survives large changes in wording (synonyms, voice, word order) but not small ones (negation, swapped roles). Counting shared words therefore fails both ways. Pre-neural systems combined word overlap, WordNet similarity and alignment features in a regressor, and Word embeddings later made soft word matching possible.
Cross-encoders and bi-encoders
A cross-encoder feeds [CLS] a [SEP] b [SEP] through one transformer, so every token of attends to every token of , and a linear head on [CLS] regresses the score. This is how BERT is fine-tuned on STS-B. It is the most accurate option, but it needs one full forward pass per pair.
A bi-encoder encodes each sentence on its own into a vector, and , and compares them with cosine similarity. Sentence-BERT (Reimers and Gurevych 2019) trains BERT this way in a Siamese setup with shared weights. How such encoders are trained is covered in Sentence embeddings and Contrastive language learning.
- pooled sentence embeddings (often the mean of the token vectors)
- gold similarity rescaled to , e.g. STS score / 5
- Sentence-BERT's regression objective; a cross-encoder instead applies MSE to the [CLS] head's output
Cost decides which to use. Finding the most similar pair among 10,000 sentences takes cross-encoder passes, about 65 hours with BERT in the Sentence-BERT paper, against about 5 seconds with a bi-encoder. The usual pattern is to retrieve candidates with a bi-encoder and re-score the top few with a cross-encoder.
Evaluation
STS systems are scored by how well their scores correlate with the human ones. Spearman's is Pearson's correlation computed on ranks. It only asks whether the model orders the pairs correctly, so any monotone rescaling of the cosine leaves it unchanged. Sentence-BERT and most embedding papers report it.
- difference between the rank of pair i by gold score and by model score
- number of pairs (the formula assumes no ties; with ties, use Pearson on average ranks)
Datasets and reference numbers
- STS-B
- Cer, Diab, Agirre, Lopez-Gazpio, Specia 2017: 8,628 pairs from news, captions and forums (5,749 train / 1,500 dev / 1,379 test), gold 0–5. Part of GLUE.
- MRPC
- Dolan and Brockett 2005: 5,801 news sentence pairs labelled paraphrase or not, about two-thirds positive. Reported as F1 and accuracy.
- QQP
- Quora Question Pairs: over 400,000 question pairs labelled duplicate or not. Reported as F1 and accuracy in GLUE.
- PAWS
- Zhang, Baldridge, He 2019: pairs made by swapping words, so lexical overlap is high whatever the label. Bag-of-words models fail on it.
- Cross-encoder
- BERT-large fine-tuned on STS-B: 86.5 Spearman on the GLUE test set (Devlin et al. 2019).
- Bi-encoder, no STS training
- STS-B test Spearman ×100 (Reimers and Gurevych 2019): averaged GloVe 58.02, averaged BERT token vectors 46.35, SBERT-NLI-base 77.03.
The last row is the surprise. Averaging raw BERT outputs is worse than averaging GloVe vectors. Raw contextual embeddings are anisotropic: they share a dominant direction, so every pair has a high cosine (Ethayarajh 2019). Training for similarity, or removing the shared component, spreads them out.
Worked example: Spearman's ρ by hand
| pair | gold | cos | gold rank | cos rank | d² |
|---|---|---|---|---|---|
| plane ↔ airplane | 5.0 | 0.932 | 10 | 7 | 9 |
| playing ↔ plays guitar | 4.8 | 0.873 | 9 | 5 | 16 |
| slicing ↔ cutting onion | 4.6 | 0.994 | 8 | 9 | 1 |
| stocks fell ↔ market dropped | 4.2 | 0.937 | 7 | 8 | 1 |
| cat sits ↔ cat sleeping | 3.4 | 0.775 | 6 | 2 | 16 |
| reading ↔ writing a book | 2.6 | 0.910 | 5 | 6 | 1 |
| dog bites man ↔ reversed | 1.8 | 1.000 | 4 | 10 | 36 |
| boy/horse ↔ girl/bike | 1.6 | 0.830 | 3 | 4 | 1 |
| woman dancing ↔ man talking | 0.8 | 0.803 | 2 | 3 | 1 |
| child pasta ↔ sun sea | 0.2 | 0.322 | 1 | 1 | 0 |
, so . The word-order pair alone contributes 36 of the 82. Mean pooling cannot see order, so this pair gets the top cosine while its gold rank is 4.
1from scipy.stats import spearmanr2from sentence_transformers import CrossEncoder, SentenceTransformer, util34a = ["a man is playing a guitar", "the dog bites the man"]5b = ["a man plays the guitar", "the man bites the dog"]67bi = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")8u = bi.encode(a, normalize_embeddings=True) # one pass per sentence9v = bi.encode(b, normalize_embeddings=True)10print(util.cos_sim(u, v).diagonal()) # cosine per pair1112ce = CrossEncoder("cross-encoder/stsb-roberta-base")13print(ce.predict(list(zip(a, b)))) # one pass per pair1415gold = [5.0, 4.8, 4.6, 4.2, 3.4, 2.6, 1.8, 1.6, 0.8, 0.2]16cos = [0.932, 0.873, 0.994, 0.937, 0.775, 0.910, 1.000, 0.830, 0.803, 0.322]17print(round(spearmanr(gold, cos).correlation, 3)) # 0.503Use when
- +Bi-encoder: embed the corpus once, then search millions of items with nearest-neighbour indexes
- +Cross-encoder: full token-level interaction, the most accurate score per pair
- +Spearman ρ is robust to monotone rescaling, so models with different score ranges compare fairly
Avoid when
- −Bi-encoder: one vector per sentence loses detail such as word order, negation and who did what
- −Cross-encoder: passes to compare sentences; no reusable vectors
- −Gold STS ratings are noisy and domain-specific; scores on STS-B do not guarantee good retrieval
Check yourself
You must find near-duplicates among 1 million support tickets. Which design fits?
Check yourself
Model A's cosines are exactly Model B's cosines squared (all positive). How do their STS-B Spearman scores compare?
Remember
- ✓STS: graded 0–5 similarity of a sentence pair; paraphrase detection is the binary version (MRPC, QQP).
- ✓Cross-encoder: one joint pass per pair, most accurate. Bi-encoder: , one pass per sentence, searchable.
- ✓Evaluate with Spearman : rank agreement with human scores.
- ✓Mean pooling ignores word order; raw contextual embeddings are anisotropic and need similarity training.
- ✓Retrieve with a bi-encoder, re-rank with a cross-encoder; tune thresholds per model.