Sentence embeddings

One fixed-size vector per sentence, compared by cosine. Sentence-BERT (2019) fine-tunes a siamese BERT with mean pooling, so 10,000 sentences are embedded once instead of ~50M cross-encoder passes.

8 min read723 wordscore0 viewsupdated 2026-10-15

Definition

A sentence embedding is a function f:text→Rdf:\text{text}\to\mathbb{R}^d that maps a sentence (or passage) to one vector, trained so that cos⁡(f(a),f(b))\cos(f(a),f(b)) is high when aa and bb mean the same thing. The modern recipe is Sentence-BERT (Reimers and Gurevych, EMNLP 2019): a pretrained transformer, a pooling step over its token states, and fine-tuning in a siamese setup on sentence pairs.

The input is a variable-length token sequence; the output is a fixed dd, typically 384 to 1024. The vector must be useful without the other sentence in view: you embed a million documents once, store the vectors, and compare any query against them by a dot product.

That is what makes it hard. A cross-encoder (see Semantic similarity) reads both sentences together and can attend across them. A sentence embedding must compress everything a future comparison could need into dd numbers, before it knows the comparison.

Before transformers the baselines were averaged word vectors (see Word embeddings, GloVe) and their weighted variant SIF (Arora, Liang and Ma, ICLR 2017), which weights each word by a/(a+p(w))a/(a+p(w)) and removes the first principal component. Trained encoders followed: Skip-Thought (Kiros et al., 2015) predicted neighbouring sentences; InferSent (Conneau et al., EMNLP 2017) trained a BiLSTM with max pooling on SNLI; the Universal Sentence Encoder (Cer et al., 2018) mixed several tasks.

Sentence-BERT: siamese encoder plus pooling

Raw BERT is a poor sentence encoder. Its [CLS] vector is trained only for next-sentence prediction, and averaged token states inherit the anisotropy described in Contextual embeddings. SBERT fixes this by fine-tuning: both sentences of a pair go through the same network, their token states are pooled, and a loss on the two vectors is backpropagated into BERT.

u=∑i=1nmi hi∑i=1nmi,y^=softmax⁡ ⁣(W [ u;  v;  ∣u−v∣ ]),W∈R3×3du=\frac{\sum_{i=1}^{n} m_i\,h_i}{\sum_{i=1}^{n} m_i},\qquad \hat y=\operatorname{softmax}\!\big(W\,[\,u;\;v;\;|u-v|\,]\big),\qquad W\in\mathbb{R}^{3\times 3d}
hih_i
final hidden state of token ii (including [CLS] and [SEP])
mim_i
attention mask: 1 for real tokens, 0 for padding, so padding never enters the mean
[u;v;∣u−v∣][u;v;|u-v|]
concatenation fed to a 3-way NLI classifier (entailment, neutral, contradiction) during training only

The paper trained on SNLI (570k pairs) plus MultiNLI (430k) for one epoch, batch size 16, Adam with learning rate 2×10−52\times10^{-5} and 10% warm-up. Mean pooling was the default and slightly beat [CLS] and max pooling. In the ablation the element-wise difference ∣u−v∣|u-v| was the most important part of the classifier input. A regression variant instead trains cos⁡(u,v)\cos(u,v) against gold STS scores with a mean-squared error.

“A man is playing a guitar.”9 WordPiece tokens incl. [CLS] [SEP]BERT encoder (12 layers)one forward pass per sentenceh11h21h31h41h51h61h71h81h91mean pool Σ mᵢhᵢ / Σ mᵢsum of 9 states / 9u ∈ ℝ^768“Someone plays the guitar.”7 WordPiece tokens incl. [CLS] [SEP], + 2 [PAD]BERT encoder (12 layers)one forward pass per sentenceh11h21h31h41h51h61h71h80h90mean pool Σ mᵢhᵢ / Σ mᵢsum of 7 states / 7v ∈ ℝ^768tiedmasktraining (NLI classification)softmax(W [u; v; |u − v|]), W ∈ ℝ^(3×2304)label: entailment / neutral / contradictioninference (head discarded)cos(u, v) = u·v / (‖u‖ ‖v‖)embed n sentences once, compare all pairs
fig.Sentence-BERT. Both sentences go through one shared BERT; the 9 and 7 real token states are averaged under the attention mask (the two [PAD] states are excluded). The NLI classifier on [u; v; |u − v|] exists only during training. At inference each sentence is embedded once and pairs are compared by cosine.

The payoff is cost. Finding the most similar pair among 10,000 sentences takes (1042)≈5×107\binom{10^4}{2}\approx 5\times10^7 cross-encoder passes, about 65 hours with BERT in the paper's estimate. SBERT embeds the 10,000 sentences once and compares them with one matrix product: about 5 seconds.

Normalise, then cosine

cos⁡(u,v)=u⊤v∥u∥ ∥v∥,∥u^−v^∥2=2−2cos⁡(u,v)for unit u^,v^\cos(u,v)=\frac{u^\top v}{\lVert u\rVert\,\lVert v\rVert},\qquad \lVert \hat u-\hat v\rVert^2 = 2-2\cos(u,v)\quad\text{for unit } \hat u,\hat v
u^\hat u
u/∥u∥u/\lVert u\rVert; after L2-normalisation, dot product, cosine and Euclidean distance give the same ranking

So vector databases store unit vectors and run maximum inner product search. Cosine values are only comparable within one model: 0.6 may be a strong match in one model and noise in another, because each has its own baseline similarity between unrelated sentences.

EncoderPooling / trainingAvg STS
BERT [CLS]pretrained only29.19
Averaged BERT statespretrained only54.81
Averaged GloVestatic word vectors61.32
InferSent (GloVe)BiLSTM + max pool, SNLI65.01
Universal Sentence Encodertransformer, multi-task71.22
SBERT-NLI-basemean pool, SNLI + MultiNLI74.89
SBERT-NLI-largemean pool, SNLI + MultiNLI76.55
Unsupervised SimCSEdropout positives, Wikipedia76.3
Supervised SimCSENLI positives + hard negatives81.6
Average Spearman ρ × 100 on seven STS tasks without STS training data. First seven rows: Reimers & Gurevych (2019), Table 1; SimCSE rows: Gao et al. (2021), BERT-base

Pretrained BERT states, averaged, do worse than averaged GloVe vectors. Fine-tuning for the geometry, not model size, is what makes a sentence encoder. The contrastive objectives behind the SimCSE rows are the subject of Contrastive language learning.

mean pooling · offset A = 1.5 · raw · cosine similarityevery token vector = word vector + A·u (one shared direction u: anisotropy)a man plays the guitars1someone is playing a guitars2the cat sleeps on the sofas3a kitten naps on the couchs4stocks fell sharply this mornings5markets dropped todays610.980.960.950.920.880.9810.950.950.940.900.960.9510.990.940.910.950.950.9910.940.940.920.940.940.9410.960.880.900.910.940.961outlined = paraphrase pairparaphrase mean0.976unrelated mean0.931gap 0.045nearest neighbouris the paraphrase:6 / 6rule cos ≥ 0.50pairs right:3 / 15the shared offset squeezes every cosine toward 1: ranking survives, absolute thresholds do not
poolingpost-process
fig.Six toy sentences, three paraphrase pairs. Every token vector gets the same offset A·u. At the defaults (mean pooling, A = 1.5, raw) paraphrases average 0.976 and unrelated pairs 0.931: the gap is 0.045, and the rule cos ≥ 0.5 gets only 3 of 15 pairs right, although the nearest neighbour is still the paraphrase 6 of 6 times. Press centred: the means become 0.579 and −0.369 and the rule gets 14 of 15. Set A = 0 to see raw cosines of 0.700 vs 0.047. Switch to [CLS]: an untuned [CLS] finds the paraphrase at most 1 time in 6, centred or not.

Worked example: masked mean pooling and cosine

Sentence A has three token states (1.0,0.8,0.2)(1.0,0.8,0.2), (0.9,0.1,0.6)(0.9,0.1,0.6), (1.1,0.3,0.1)(1.1,0.3,0.1). Sentence B has two real states (1.0,0.7,0.3)(1.0,0.7,0.3), (1.2,0.2,0.4)(1.2,0.2,0.4) and one padding state (0.0,2.0,−1.0)(0.0,2.0,-1.0) with m=0m=0. An unrelated sentence has c=(0.9,−0.2,0.9)c=(0.9,-0.2,0.9).

QuantityComputationValue
uumean of A's 3 states(1.0, 0.4, 0.3)
vvmean of B's 2 real states(1.1, 0.45, 0.35)
cos⁡(u,v)\cos(u,v)1.385/(1.1180⋅1.2390)1.385/(1.1180\cdot1.2390)0.9999
vv with [PAD] averaged inmean of all 3 states(0.7333, 0.9667, −0.1)
cos⁡(u,vpad)\cos(u,v_{\text{pad}})padding leaks into the vector0.8008
cos⁡(u,c)\cos(u,c)unrelated sentence, raw0.7567
after centring on μ=(1.0, 0.2167, 0.5167)\mu=(1.0,\,0.2167,\,0.5167)cos⁡(u−μ, v−μ)\cos(u-\mu,\,v-\mu) and cos⁡(u−μ, c−μ)\cos(u-\mu,\,c-\mu)0.9153 and −0.9771
Every value computed in numpy (4 decimals)

Two lessons. Ignoring the mask changed a near-perfect match (0.9999) into 0.8008, and the result would depend on how much padding the batch happened to need. And a raw cosine of 0.7567 between unrelated sentences only means something relative to the model's baseline; subtracting the mean exposes it. With three vectors the centring is exaggerated, but the direction of the effect is general.

python
1from sentence_transformers import SentenceTransformer, util
2import torch
3from transformers import AutoTokenizer, AutoModel
4
5docs = ["A man is playing a guitar.", "The cat sleeps on the sofa.", "Stocks fell sharply."]
6model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") # 6 layers, 384-d
7E = model.encode(docs, normalize_embeddings=True) # (3, 384), unit rows
8q = model.encode("Someone plays the guitar.", normalize_embeddings=True)
9print(util.cos_sim(q, E)) # dot = cosine here
10
11# what the pooling layer does under the hood
12tok = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
13enc = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
14batch = tok(docs, padding=True, return_tensors="pt")
15with torch.no_grad():
16 H = enc(**batch).last_hidden_state # (3, n, 384)
17m = batch["attention_mask"].unsqueeze(-1).float() # 0 on [PAD]
18u = (H * m).sum(1) / m.sum(1) # masked mean
19u = torch.nn.functional.normalize(u, dim=-1)
Embed, normalise, rank (sentence-transformers), and the same mean pooling by hand

Evaluation and variants

STS
Spearman correlation between cos⁡(u,v)\cos(u,v) and human 0–5 similarity ratings (STS 2012–2016, STS-B, SICK-R)
MTEB
Muennighoff et al. (EACL 2023): 8 task types (bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, summarisation), 58 datasets, 112 languages; no single model won every task type
Matryoshka
Kusupati et al. (NeurIPS 2022): the loss is summed over nested prefixes of the vector, so the first 64 or 128 dimensions are themselves a usable, cheaper embedding
Asymmetric search
Retrieval models often embed queries and passages differently (e.g. E5's query: / passage: prefixes); see Retrieval-augmented generation (RAG)

Check yourself

You batch one short and one long sentence and mean-pool all token states, padding included. What happens?

Check yourself

Two L2-normalised embeddings have cosine 0.8. What is their squared Euclidean distance?

Remember

  • ✓A sentence embedding is computed once per text; similarity is then a dot product, so nn texts cost nn encoder passes instead of (n2)\binom{n}{2}.
  • ✓SBERT: shared BERT, masked mean pooling u=∑mihi/∑miu=\sum m_ih_i/\sum m_i, trained with softmax(W[u;v;∣u−v∣])(W[u;v;|u-v|]) on NLI; 74.89 avg STS vs 29.19 for raw [CLS].
  • ✓Normalise to unit length: cosine, dot product and ∥u^−v^∥2=2−2cos⁡\lVert\hat u-\hat v\rVert^2=2-2\cos then rank alike.
  • ✓Anisotropy inflates every cosine; rankings may survive, fixed thresholds do not. Centring, whitening or contrastive training spreads the space.
  • ✓Benchmark on MTEB-style task mixes; STS alone does not predict retrieval or clustering quality.
← back to Deep Learning roadmap