Definition
A sentence embedding is a function that maps a sentence (or passage) to one vector, trained so that is high when and mean the same thing. The modern recipe is Sentence-BERT (Reimers and Gurevych, EMNLP 2019): a pretrained transformer, a pooling step over its token states, and fine-tuning in a siamese setup on sentence pairs.
The input is a variable-length token sequence; the output is a fixed , typically 384 to 1024. The vector must be useful without the other sentence in view: you embed a million documents once, store the vectors, and compare any query against them by a dot product.
That is what makes it hard. A cross-encoder (see Semantic similarity) reads both sentences together and can attend across them. A sentence embedding must compress everything a future comparison could need into numbers, before it knows the comparison.
Before transformers the baselines were averaged word vectors (see Word embeddings, GloVe) and their weighted variant SIF (Arora, Liang and Ma, ICLR 2017), which weights each word by and removes the first principal component. Trained encoders followed: Skip-Thought (Kiros et al., 2015) predicted neighbouring sentences; InferSent (Conneau et al., EMNLP 2017) trained a BiLSTM with max pooling on SNLI; the Universal Sentence Encoder (Cer et al., 2018) mixed several tasks.
Sentence-BERT: siamese encoder plus pooling
Raw BERT is a poor sentence encoder. Its [CLS] vector is trained only for next-sentence prediction, and averaged token states inherit the anisotropy described in Contextual embeddings. SBERT fixes this by fine-tuning: both sentences of a pair go through the same network, their token states are pooled, and a loss on the two vectors is backpropagated into BERT.
- final hidden state of token (including
[CLS]and[SEP]) - attention mask: 1 for real tokens, 0 for padding, so padding never enters the mean
- concatenation fed to a 3-way NLI classifier (entailment, neutral, contradiction) during training only
The paper trained on SNLI (570k pairs) plus MultiNLI (430k) for one epoch, batch size 16, Adam with learning rate and 10% warm-up. Mean pooling was the default and slightly beat [CLS] and max pooling. In the ablation the element-wise difference was the most important part of the classifier input. A regression variant instead trains against gold STS scores with a mean-squared error.
The payoff is cost. Finding the most similar pair among 10,000 sentences takes cross-encoder passes, about 65 hours with BERT in the paper's estimate. SBERT embeds the 10,000 sentences once and compares them with one matrix product: about 5 seconds.
Normalise, then cosine
- ; after L2-normalisation, dot product, cosine and Euclidean distance give the same ranking
So vector databases store unit vectors and run maximum inner product search. Cosine values are only comparable within one model: 0.6 may be a strong match in one model and noise in another, because each has its own baseline similarity between unrelated sentences.
| Encoder | Pooling / training | Avg STS |
|---|---|---|
| BERT [CLS] | pretrained only | 29.19 |
| Averaged BERT states | pretrained only | 54.81 |
| Averaged GloVe | static word vectors | 61.32 |
| InferSent (GloVe) | BiLSTM + max pool, SNLI | 65.01 |
| Universal Sentence Encoder | transformer, multi-task | 71.22 |
| SBERT-NLI-base | mean pool, SNLI + MultiNLI | 74.89 |
| SBERT-NLI-large | mean pool, SNLI + MultiNLI | 76.55 |
| Unsupervised SimCSE | dropout positives, Wikipedia | 76.3 |
| Supervised SimCSE | NLI positives + hard negatives | 81.6 |
Pretrained BERT states, averaged, do worse than averaged GloVe vectors. Fine-tuning for the geometry, not model size, is what makes a sentence encoder. The contrastive objectives behind the SimCSE rows are the subject of Contrastive language learning.
Worked example: masked mean pooling and cosine
Sentence A has three token states , , . Sentence B has two real states , and one padding state with . An unrelated sentence has .
| Quantity | Computation | Value |
|---|---|---|
| mean of A's 3 states | (1.0, 0.4, 0.3) | |
| mean of B's 2 real states | (1.1, 0.45, 0.35) | |
| 0.9999 | ||
| with [PAD] averaged in | mean of all 3 states | (0.7333, 0.9667, −0.1) |
| padding leaks into the vector | 0.8008 | |
| unrelated sentence, raw | 0.7567 | |
| after centring on | and | 0.9153 and −0.9771 |
Two lessons. Ignoring the mask changed a near-perfect match (0.9999) into 0.8008, and the result would depend on how much padding the batch happened to need. And a raw cosine of 0.7567 between unrelated sentences only means something relative to the model's baseline; subtracting the mean exposes it. With three vectors the centring is exaggerated, but the direction of the effect is general.
1from sentence_transformers import SentenceTransformer, util2import torch3from transformers import AutoTokenizer, AutoModel45docs = ["A man is playing a guitar.", "The cat sleeps on the sofa.", "Stocks fell sharply."]6model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") # 6 layers, 384-d7E = model.encode(docs, normalize_embeddings=True) # (3, 384), unit rows8q = model.encode("Someone plays the guitar.", normalize_embeddings=True)9print(util.cos_sim(q, E)) # dot = cosine here1011# what the pooling layer does under the hood12tok = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")13enc = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")14batch = tok(docs, padding=True, return_tensors="pt")15with torch.no_grad():16 H = enc(**batch).last_hidden_state # (3, n, 384)17m = batch["attention_mask"].unsqueeze(-1).float() # 0 on [PAD]18u = (H * m).sum(1) / m.sum(1) # masked mean19u = torch.nn.functional.normalize(u, dim=-1)Evaluation and variants
- STS
- Spearman correlation between and human 0–5 similarity ratings (STS 2012–2016, STS-B, SICK-R)
- MTEB
- Muennighoff et al. (EACL 2023): 8 task types (bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, summarisation), 58 datasets, 112 languages; no single model won every task type
- Matryoshka
- Kusupati et al. (NeurIPS 2022): the loss is summed over nested prefixes of the vector, so the first 64 or 128 dimensions are themselves a usable, cheaper embedding
- Asymmetric search
- Retrieval models often embed queries and passages differently (e.g. E5's
query:/passage:prefixes); see Retrieval-augmented generation (RAG)
Check yourself
You batch one short and one long sentence and mean-pool all token states, padding included. What happens?
Check yourself
Two L2-normalised embeddings have cosine 0.8. What is their squared Euclidean distance?
Remember
- ✓A sentence embedding is computed once per text; similarity is then a dot product, so texts cost encoder passes instead of .
- ✓SBERT: shared BERT, masked mean pooling , trained with softmax on NLI; 74.89 avg STS vs 29.19 for raw
[CLS]. - ✓Normalise to unit length: cosine, dot product and then rank alike.
- ✓Anisotropy inflates every cosine; rankings may survive, fixed thresholds do not. Centring, whitening or contrastive training spreads the space.
- ✓Benchmark on MTEB-style task mixes; STS alone does not predict retrieval or clustering quality.