# Sentence embeddings

> One fixed-size vector per sentence, compared by cosine. Sentence-BERT (2019) fine-tunes a siamese BERT with mean pooling, so 10,000 sentences are embedded once instead of ~50M cross-encoder passes.

Source: https://www.mlfoundry.in/learn/sentence-embeddings  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 8 min read · updated 2026-10-15

**A sentence embedding:** is a function $f:\text{text}\to\mathbb{R}^d$ that maps a sentence (or passage) to one vector, trained so that $\cos(f(a),f(b))$ is high when $a$ and $b$ mean the same thing. The modern recipe is Sentence-BERT (Reimers and Gurevych, EMNLP 2019): a pretrained transformer, a pooling step over its token states, and fine-tuning in a siamese setup on sentence pairs.

The input is a variable-length token sequence; the output is a fixed $d$, typically 384 to 1024. The vector must be useful *without* the other sentence in view: you embed a million documents once, store the vectors, and compare any query against them by a dot product.

That is what makes it hard. A cross-encoder (see **Semantic similarity**) reads both sentences together and can attend across them. A sentence embedding must compress everything a future comparison could need into $d$ numbers, before it knows the comparison.

Before transformers the baselines were averaged word vectors (see **Word embeddings**, **GloVe**) and their weighted variant SIF (Arora, Liang and Ma, ICLR 2017), which weights each word by $a/(a+p(w))$ and removes the first principal component. Trained encoders followed: Skip-Thought (Kiros et al., 2015) predicted neighbouring sentences; InferSent (Conneau et al., EMNLP 2017) trained a BiLSTM with max pooling on SNLI; the Universal Sentence Encoder (Cer et al., 2018) mixed several tasks.

## Sentence-BERT: siamese encoder plus pooling

Raw BERT is a poor sentence encoder. Its `[CLS]` vector is trained only for next-sentence prediction, and averaged token states inherit the anisotropy described in **Contextual embeddings**. SBERT fixes this by fine-tuning: both sentences of a pair go through the *same* network, their token states are pooled, and a loss on the two vectors is backpropagated into BERT.

$$
u=\frac{\sum_{i=1}^{n} m_i\,h_i}{\sum_{i=1}^{n} m_i},\qquad \hat y=\operatorname{softmax}\!\big(W\,[\,u;\;v;\;|u-v|\,]\big),\qquad W\in\mathbb{R}^{3\times 3d}
$$

- $h_i$: final hidden state of token $i$ (including `[CLS]` and `[SEP]`)
- $m_i$: attention mask: 1 for real tokens, 0 for padding, so padding never enters the mean
- $[u;v;|u-v|]$: concatenation fed to a 3-way NLI classifier (entailment, neutral, contradiction) during training only

The paper trained on SNLI (570k pairs) plus MultiNLI (430k) for one epoch, batch size 16, Adam with learning rate $2\times10^{-5}$ and 10% warm-up. Mean pooling was the default and slightly beat `[CLS]` and max pooling. In the ablation the element-wise difference $|u-v|$ was the most important part of the classifier input. A regression variant instead trains $\cos(u,v)$ against gold STS scores with a mean-squared error.

> Figure: Sentence-BERT. Both sentences go through one shared BERT; the 9 and 7 real token states are averaged under the attention mask (the two [PAD] states are excluded). The NLI classifier on [u; v; |u − v|] exists only during training. At inference each sentence is embedded once and pairs are compared by cosine.

The payoff is cost. Finding the most similar pair among 10,000 sentences takes $\binom{10^4}{2}\approx 5\times10^7$ cross-encoder passes, about 65 hours with BERT in the paper's estimate. SBERT embeds the 10,000 sentences once and compares them with one matrix product: about 5 seconds.

## Normalise, then cosine

$$
\cos(u,v)=\frac{u^\top v}{\lVert u\rVert\,\lVert v\rVert},\qquad \lVert \hat u-\hat v\rVert^2 = 2-2\cos(u,v)\quad\text{for unit } \hat u,\hat v
$$

- $\hat u$: $u/\lVert u\rVert$; after L2-normalisation, dot product, cosine and Euclidean distance give the same ranking

So vector databases store unit vectors and run maximum inner product search. Cosine values are only comparable *within* one model: 0.6 may be a strong match in one model and noise in another, because each has its own baseline similarity between unrelated sentences.

Average Spearman ρ × 100 on seven STS tasks without STS training data. First seven rows: Reimers & Gurevych (2019), Table 1; SimCSE rows: Gao et al. (2021), BERT-base

| Encoder | Pooling / training | Avg STS |
| --- | --- | --- |
| BERT [CLS] | pretrained only | 29.19 |
| Averaged BERT states | pretrained only | 54.81 |
| Averaged GloVe | static word vectors | 61.32 |
| InferSent (GloVe) | BiLSTM + max pool, SNLI | 65.01 |
| Universal Sentence Encoder | transformer, multi-task | 71.22 |
| SBERT-NLI-base | mean pool, SNLI + MultiNLI | 74.89 |
| SBERT-NLI-large | mean pool, SNLI + MultiNLI | 76.55 |
| Unsupervised SimCSE | dropout positives, Wikipedia | 76.3 |
| Supervised SimCSE | NLI positives + hard negatives | 81.6 |

Pretrained BERT states, averaged, do *worse* than averaged GloVe vectors. Fine-tuning for the geometry, not model size, is what makes a sentence encoder. The contrastive objectives behind the SimCSE rows are the subject of **Contrastive language learning**.

> Figure: Six toy sentences, three paraphrase pairs. Every token vector gets the same offset A·u. At the defaults (mean pooling, A = 1.5, raw) paraphrases average 0.976 and unrelated pairs 0.931: the gap is 0.045, and the rule cos ≥ 0.5 gets only 3 of 15 pairs right, although the nearest neighbour is still the paraphrase 6 of 6 times. Press centred: the means become 0.579 and −0.369 and the rule gets 14 of 15. Set A = 0 to see raw cosines of 0.700 vs 0.047. Switch to [CLS]: an untuned [CLS] finds the paraphrase at most 1 time in 6, centred or not.

## Worked example: masked mean pooling and cosine

Sentence A has three token states $(1.0,0.8,0.2)$, $(0.9,0.1,0.6)$, $(1.1,0.3,0.1)$. Sentence B has two real states $(1.0,0.7,0.3)$, $(1.2,0.2,0.4)$ and one padding state $(0.0,2.0,-1.0)$ with $m=0$. An unrelated sentence has $c=(0.9,-0.2,0.9)$.

Every value computed in numpy (4 decimals)

| Quantity | Computation | Value |
| --- | --- | --- |
| $u$ | mean of A's 3 states | (1.0, 0.4, 0.3) |
| $v$ | mean of B's 2 real states | (1.1, 0.45, 0.35) |
| $\cos(u,v)$ | $1.385/(1.1180\cdot1.2390)$ | 0.9999 |
| $v$ with [PAD] averaged in | mean of all 3 states | (0.7333, 0.9667, −0.1) |
| $\cos(u,v_{\text{pad}})$ | padding leaks into the vector | 0.8008 |
| $\cos(u,c)$ | unrelated sentence, raw | 0.7567 |
| after centring on $\mu=(1.0,\,0.2167,\,0.5167)$ | $\cos(u-\mu,\,v-\mu)$ and $\cos(u-\mu,\,c-\mu)$ | 0.9153 and −0.9771 |

Two lessons. Ignoring the mask changed a near-perfect match (0.9999) into 0.8008, and the result would depend on how much padding the batch happened to need. And a raw cosine of 0.7567 between unrelated sentences only means something relative to the model's baseline; subtracting the mean exposes it. With three vectors the centring is exaggerated, but the direction of the effect is general.

```python
from sentence_transformers import SentenceTransformer, util
import torch
from transformers import AutoTokenizer, AutoModel

docs = ["A man is playing a guitar.", "The cat sleeps on the sofa.", "Stocks fell sharply."]
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")    # 6 layers, 384-d
E = model.encode(docs, normalize_embeddings=True)                       # (3, 384), unit rows
q = model.encode("Someone plays the guitar.", normalize_embeddings=True)
print(util.cos_sim(q, E))                                               # dot = cosine here

# what the pooling layer does under the hood
tok = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
enc = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
batch = tok(docs, padding=True, return_tensors="pt")
with torch.no_grad():
    H = enc(**batch).last_hidden_state                                  # (3, n, 384)
m = batch["attention_mask"].unsqueeze(-1).float()                       # 0 on [PAD]
u = (H * m).sum(1) / m.sum(1)                                           # masked mean
u = torch.nn.functional.normalize(u, dim=-1)
```

Embed, normalise, rank (sentence-transformers), and the same mean pooling by hand

**Evaluation and variants**

- **STS:** Spearman correlation between $\cos(u,v)$ and human 0–5 similarity ratings (STS 2012–2016, STS-B, SICK-R)
- **MTEB:** Muennighoff et al. (EACL 2023): 8 task types (bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, summarisation), 58 datasets, 112 languages; no single model won every task type
- **Matryoshka:** Kusupati et al. (NeurIPS 2022): the loss is summed over nested prefixes of the vector, so the first 64 or 128 dimensions are themselves a usable, cheaper embedding
- **Asymmetric search:** Retrieval models often embed queries and passages differently (e.g. E5's `query:` / `passage:` prefixes); see **Retrieval-augmented generation (RAG)**

> **Similar is not the same as relevant:** Embeddings trained on paraphrase and NLI data score "the drug is safe" and "the drug is not safe" as close: they share topic and almost every token. Negation, numbers and entity swaps are classic blind spots. If the distinction matters, rerank the top hits with a cross-encoder.

> **Pooling is part of the model:** A checkpoint fine-tuned with mean pooling must be used with mean pooling. Taking its `[CLS]` vector, or forgetting to normalise when the model was trained on normalised vectors, silently degrades every similarity.

**Question:** You batch one short and one long sentence and mean-pool all token states, padding included. What happens?

- A. Nothing: padding states are zero vectors
- B. The short sentence's vector is pulled toward the padding states, and changes with batch composition
- C. The long sentence's vector is corrupted, not the short one
- D. Cosine similarity cancels the effect because it normalises

**Answer:** B. The short sentence's vector is pulled toward the padding states, and changes with batch composition — Padding positions still produce non-zero hidden states. Only the short sentence has padding, so only its mean is diluted, by an amount that depends on the longest sentence in the batch. Normalisation rescales the vector but cannot undo the change of direction (0.9999 → 0.8008 in the worked example).

**Question:** Two L2-normalised embeddings have cosine 0.8. What is their squared Euclidean distance?

- A. 0.2
- B. 0.4
- C. 0.64
- D. 1.6

**Answer:** B. 0.4 — $\lVert\hat u-\hat v\rVert^2=\lVert\hat u\rVert^2+\lVert\hat v\rVert^2-2\hat u^\top\hat v=2-2(0.8)=0.4$. That identity is why L2 search and cosine search rank normalised vectors identically.

**Key points**

- A sentence embedding is computed once per text; similarity is then a dot product, so $n$ texts cost $n$ encoder passes instead of $\binom{n}{2}$.
- SBERT: shared BERT, masked mean pooling $u=\sum m_ih_i/\sum m_i$, trained with softmax$(W[u;v;|u-v|])$ on NLI; 74.89 avg STS vs 29.19 for raw `[CLS]`.
- Normalise to unit length: cosine, dot product and $\lVert\hat u-\hat v\rVert^2=2-2\cos$ then rank alike.
- Anisotropy inflates every cosine; rankings may survive, fixed thresholds do not. Centring, whitening or contrastive training spreads the space.
- Benchmark on MTEB-style task mixes; STS alone does not predict retrieval or clustering quality.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
