# Contrastive language learning

> Train a text encoder so each sentence picks its own positive out of a batch: InfoNCE with in-batch negatives and temperature. SimCSE uses two dropout passes as the positive; CLIP pairs captions with images.

Source: https://www.mlfoundry.in/learn/contrastive-language-learning  
Section: Deep Learning › Deep Learning for NLP  
Level: advanced · 8 min read · updated 2026-10-15

**Contrastive language learning:** trains a text encoder $f$ on pairs $(x_i,x_i^+)$ that should match (a sentence and a perturbed copy, a paraphrase, a question and its passage, a caption and its image) by making $\operatorname{sim}(f(x_i),f(x_i^+))$ larger than the similarity of $x_i$ to every other item in the batch. The standard objective is InfoNCE with in-batch negatives; the reference methods for language are SimCSE (Gao, Yao and Chen, EMNLP 2021) and, across modalities, CLIP (Radford et al., ICML 2021).

The input is a stream of positive pairs. No labels say *how* similar two texts are; the supervision is only "these two belong together, the rest of the batch does not". The output is an embedding space in which cosine similarity is meaningful, which is exactly what **Sentence embeddings** need.

The idea is older than transformers. Siamese networks compared signatures in 1993, and **Word2Vec**'s negative sampling is a contrastive loss over words and contexts. The hard part for sentences is the positive pair. Images have meaning-preserving augmentations (crops, colour jitter); text does not, because deleting or swapping one word can flip the meaning.

## InfoNCE with in-batch negatives

$$
\ell_i=-\log\frac{e^{\operatorname{sim}(h_i,h_i^+)/\tau}}{\sum_{j=1}^{N}e^{\operatorname{sim}(h_i,h_j^+)/\tau}},\qquad \operatorname{sim}(a,b)=\frac{a^\top b}{\lVert a\rVert\lVert b\rVert}
$$

- $h_i,\,h_i^+$: embeddings of the two halves of positive pair i
- $h_j^+,\ j\neq i$: the other $N-1$ positives in the batch, reused as negatives at no extra encoding cost
- $\tau$: temperature; small τ makes the softmax sharp, so the loss concentrates on the hardest negatives

Each row is an $N$-way classification: find your own partner among $N$ candidates. The batch loss is the mean of $\ell_i$, i.e. cross-entropy on the $N\times N$ similarity matrix with the diagonal as labels. The mutual-information bound and the role of $N$ are derived in **InfoNCE loss**; the pairwise margin version is **Contrastive loss**.

## SimCSE: dropout is the augmentation

Unsupervised SimCSE feeds the same sentence through BERT twice. Dropout ($p=0.1$, already inside the network) samples a different mask $z$ and $z'$ each time, so $h_i=f(x_i;z)$ and $h_i^+=f(x_i;z')$ differ slightly but mean the same. The other sentences in the batch are the negatives. Training used $10^6$ sentences from English Wikipedia, $\tau=0.05$, and the `[CLS]` vector with an MLP during training.

In the paper's ablations this beat cropping, word deletion, synonym replacement and masking as augmentations, and beat a next-sentence objective. Turning dropout off, or reusing the same mask for both passes, made results much worse: the two views become identical and the task can be solved without learning anything about meaning.

$$
\ell_i^{\text{sup}}=-\log\frac{e^{\operatorname{sim}(h_i,h_i^+)/\tau}}{\sum_{j=1}^{N}\Big(e^{\operatorname{sim}(h_i,h_j^+)/\tau}+e^{\operatorname{sim}(h_i,h_j^-)/\tau}\Big)}
$$

- $h_i^+$: supervised SimCSE: an NLI hypothesis entailed by premise i
- $h_j^-$: a **hard negative**: the contradiction hypothesis of premise $j$. Close in wording, opposite in meaning

> Figure: One SimCSE batch. Each sentence is encoded twice with different dropout masks; the diagonal holds the positives and every off-diagonal entry is a free negative. With these example cosines and τ = 0.05, row 1 is already solved (loss 0.00004). Adding the contradiction "two dogs sleep indoors" (cosine 0.78) as a hard negative raises the loss to 0.0591: that column is where the gradient now comes from.

## Alignment and uniformity

Wang and Isola (ICML 2020) split what a contrastive loss optimises into two measurable properties of normalised embeddings:

$$
\ell_{\text{align}}=\mathbb{E}_{(x,x^+)}\big\lVert f(x)-f(x^+)\big\rVert^2,\qquad \ell_{\text{uniform}}=\log\,\mathbb{E}_{x,y}\,e^{-2\lVert f(x)-f(y)\rVert^2}
$$

- $\ell_{\text{align}}$: positives should land close: lower is better
- $\ell_{\text{uniform}}$: random pairs should spread over the sphere: lower (more negative) is better

The numerator of InfoNCE pulls toward alignment; the denominator pushes toward uniformity. Without negatives the trivial solution is perfect alignment by mapping every input to one point. Pretrained BERT is anisotropic (see **Contextual embeddings**), i.e. poorly uniform; Gao et al. showed that SimCSE mainly improves uniformity while keeping alignment.

> Figure: Unsupervised SimCSE on a tiny linear encoder whose 8 outputs live on a circle. Defaults (InfoNCE, τ = 0.1, dropout 0.1): the sentences start in a cone with mean cosine 0.817; at step 20 the cone first tightens (0.893, ℓ_align 0.029), then by step 400 it spreads to −0.077, with ℓ_uniform −0.324 → −2.022 (−2.363 is even spacing) and ℓ_align 0.096. Press positives only: mean cosine 1.000, ℓ_uniform −0.002, a collapse. Set dropout to 0: perfect uniformity (−2.363) but ℓ_align 0.405 on noisy test views.

**Reference systems**

- **SimCSE (BERT-base):** average Spearman on 7 STS tasks: 76.3 unsupervised, 81.6 supervised (NLI positives, contradiction hard negatives); $\tau=0.05$
- **DPR:** Karpukhin et al. (EMNLP 2020): question and passage BERT encoders, in-batch negatives plus one BM25 hard negative per question; see **Retrieval-augmented generation (RAG)**
- **CLIP:** Radford et al. (2021): 400M image–text pairs, batch 32,768, symmetric InfoNCE over images and captions, learnable τ initialised to 0.07; zero-shot ImageNet 76.2% for ViT-L/14@336px, matching the original supervised ResNet-50
- **Alignment / uniformity:** Wang and Isola (ICML 2020), with $t=2$ in the Gaussian potential as above

## Worked example: why hard negatives matter

One anchor has cosine 0.85 with its positive, 0.40 and 0.30 with two easy in-batch negatives, and 0.80 with a hard negative. The gradient of $\ell$ with respect to each cosine is $(p_j-\mathbb{1}[j=+])/\tau$.

Softmax weights, loss and gradient on the hard negative's cosine (computed in numpy)

| Setting | p(positive) | p(hard neg) | Loss | ∂ℓ/∂cos(hard) |
| --- | --- | --- | --- | --- |
| τ = 0.05, easy negatives only | 0.9999 | – | 0.0001 | – |
| τ = 0.05, + hard negative | 0.7310 | 0.2689 | 0.3134 | 5.378 |
| τ = 0.5, easy negatives only | 0.5749 | – | 0.5536 | – |
| τ = 0.5, + hard negative | 0.3782 | 0.3422 | 0.9724 | 0.684 |

At $\tau=0.05$ the easy negatives contribute almost nothing: their gradient is 0.0025 and 0.0003 without the hard negative, and the loss is 0.0001. The single hard negative takes a gradient of 5.378, about 3,000 times that of the easy negative at 0.40 (0.0018). At $\tau=0.5$ the weight is shared: the easy negatives still get 0.31 and 0.25. Low temperature plus hard negatives is what teaches fine distinctions.

```python
import torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
enc = AutoModel.from_pretrained("bert-base-uncased")       # hidden dropout 0.1
enc.train()                                                 # dropout ON: it is the augmentation

def simcse_loss(sentences, tau=0.05):
    batch = tok(sentences, padding=True, truncation=True, return_tensors="pt")
    h1 = enc(**batch).last_hidden_state[:, 0]               # [CLS], dropout mask z
    h2 = enc(**batch).last_hidden_state[:, 0]               # same input, new mask z'
    sim = F.cosine_similarity(h1.unsqueeze(1), h2.unsqueeze(0), dim=-1) / tau   # (N, N)
    labels = torch.arange(len(sentences))                   # positives on the diagonal
    return F.cross_entropy(sim, labels)

loss = simcse_loss(["two dogs run on grass", "a chef slices onions", "the train is late"])
loss.backward()
```

Unsupervised SimCSE loss: two forward passes in train mode, cross-entropy on the similarity matrix

> **False negatives in the batch:** In-batch negatives assume no two items in a batch match. Duplicates, paraphrases or two questions with the same answer passage get pushed apart anyway. This grows with batch size and with narrow-domain data; deduplicate, or mask out known positives in the denominator.

> **Why the batch size matters:** The negatives come free with the batch, so a bigger batch is a harder classification problem and a tighter mutual-information bound ($\log N$). This is why CLIP used 32,768 pairs per batch and why retrieval trainers share negatives across GPUs, or keep a queue of past embeddings as in MoCo.

**Strengths**

- Needs only pairs, not graded similarity labels; unsupervised SimCSE needs only raw sentences
- In-batch negatives make every encoded item serve N − 1 other rows at no extra forward pass
- Directly optimises the geometry used at inference: cosine search over normalised vectors

**Limitations**

- Quality depends on the negatives: easy ones give near-zero gradient, false ones push true matches apart
- Large batches cost memory; gradient caching or cross-device negatives add engineering
- Learns similarity, not truth: negation and number changes remain hard unless mined as hard negatives

**Question:** In unsupervised SimCSE you accidentally call model.eval() before computing both views. What happens?

- A. Nothing; dropout only matters at inference
- B. The two views are identical, the positive is trivially the best match, and the encoder learns little about meaning
- C. Training diverges because τ is too small
- D. The model becomes a supervised SimCSE model

**Answer:** B. The two views are identical, the positive is trivially the best match, and the encoder learns little about meaning — `eval()` disables dropout, so $f(x;z)=f(x;z')$. Each row's positive has cosine 1, which no negative can beat; the loss falls by spreading the batch apart without any pressure to map meaning-preserving perturbations together. The paper's no-dropout and same-mask ablations were much worse.

**Question:** Lowering τ from 0.5 to 0.05 mainly changes which negatives drive the gradient. How?

- A. All negatives get equal weight
- B. The gradient concentrates on the negatives most similar to the anchor
- C. Only the positive receives gradient
- D. Easy negatives get more weight than hard ones

**Answer:** B. The gradient concentrates on the negatives most similar to the anchor — Negative $j$'s gradient is $p_j/\tau$ with $p_j\propto e^{\operatorname{sim}_j/\tau}$. A smaller τ makes the softmax sharper, so weight moves to the highest-similarity negatives: in the worked example the hard negative's gradient is 5.378 vs 0.0018 for an easy one at τ = 0.05.

**Key points**

- InfoNCE on an $N\times N$ cosine matrix: diagonal = positives, off-diagonal = free in-batch negatives, temperature $\tau$.
- SimCSE: the positive is the same sentence under a second dropout mask (p = 0.1, τ = 0.05); supervised SimCSE adds NLI entailment positives and contradiction hard negatives.
- Alignment pulls positives together; uniformity, $\log\mathbb{E}e^{-2\lVert f(x)-f(y)\rVert^2}$, spreads the space. Without negatives the encoder collapses.
- Easy negatives give almost no gradient at small τ; hard negatives carry the signal.
- CLIP is the same objective across modalities: images vs captions, batch 32,768, learnable τ.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
