Contrastive language learning

Train a text encoder so each sentence picks its own positive out of a batch: InfoNCE with in-batch negatives and temperature. SimCSE uses two dropout passes as the positive; CLIP pairs captions with images.

8 min read690 wordsadvanced0 viewsupdated 2026-10-15

Definition

Contrastive language learning trains a text encoder ff on pairs (xi,xi+)(x_i,x_i^+) that should match (a sentence and a perturbed copy, a paraphrase, a question and its passage, a caption and its image) by making sim⁡(f(xi),f(xi+))\operatorname{sim}(f(x_i),f(x_i^+)) larger than the similarity of xix_i to every other item in the batch. The standard objective is InfoNCE with in-batch negatives; the reference methods for language are SimCSE (Gao, Yao and Chen, EMNLP 2021) and, across modalities, CLIP (Radford et al., ICML 2021).

The input is a stream of positive pairs. No labels say how similar two texts are; the supervision is only "these two belong together, the rest of the batch does not". The output is an embedding space in which cosine similarity is meaningful, which is exactly what Sentence embeddings need.

The idea is older than transformers. Siamese networks compared signatures in 1993, and Word2Vec's negative sampling is a contrastive loss over words and contexts. The hard part for sentences is the positive pair. Images have meaning-preserving augmentations (crops, colour jitter); text does not, because deleting or swapping one word can flip the meaning.

InfoNCE with in-batch negatives

ℓi=−log⁡esim⁡(hi,hi+)/τ∑j=1Nesim⁡(hi,hj+)/τ,sim⁡(a,b)=a⊤b∥a∥∥b∥\ell_i=-\log\frac{e^{\operatorname{sim}(h_i,h_i^+)/\tau}}{\sum_{j=1}^{N}e^{\operatorname{sim}(h_i,h_j^+)/\tau}},\qquad \operatorname{sim}(a,b)=\frac{a^\top b}{\lVert a\rVert\lVert b\rVert}
hi, hi+h_i,\,h_i^+
embeddings of the two halves of positive pair i
hj+, j≠ih_j^+,\ j\neq i
the other N−1N-1 positives in the batch, reused as negatives at no extra encoding cost
τ\tau
temperature; small τ makes the softmax sharp, so the loss concentrates on the hardest negatives

Each row is an NN-way classification: find your own partner among NN candidates. The batch loss is the mean of ℓi\ell_i, i.e. cross-entropy on the N×NN\times N similarity matrix with the diagonal as labels. The mutual-information bound and the role of NN are derived in InfoNCE loss; the pairwise margin version is Contrastive loss.

SimCSE: dropout is the augmentation

Unsupervised SimCSE feeds the same sentence through BERT twice. Dropout (p=0.1p=0.1, already inside the network) samples a different mask zz and z′z' each time, so hi=f(xi;z)h_i=f(x_i;z) and hi+=f(xi;z′)h_i^+=f(x_i;z') differ slightly but mean the same. The other sentences in the batch are the negatives. Training used 10610^6 sentences from English Wikipedia, τ=0.05\tau=0.05, and the [CLS] vector with an MLP during training.

In the paper's ablations this beat cropping, word deletion, synonym replacement and masking as augmentations, and beat a next-sentence objective. Turning dropout off, or reusing the same mask for both passes, made results much worse: the two views become identical and the task can be solved without learning anything about meaning.

ℓisup=−log⁡esim⁡(hi,hi+)/τ∑j=1N(esim⁡(hi,hj+)/τ+esim⁡(hi,hj−)/τ)\ell_i^{\text{sup}}=-\log\frac{e^{\operatorname{sim}(h_i,h_i^+)/\tau}}{\sum_{j=1}^{N}\Big(e^{\operatorname{sim}(h_i,h_j^+)/\tau}+e^{\operatorname{sim}(h_i,h_j^-)/\tau}\Big)}
hi+h_i^+
supervised SimCSE: an NLI hypothesis entailed by premise i
hj−h_j^-
a hard negative: the contradiction hypothesis of premise jj. Close in wording, opposite in meaning
SimCSE: the same sentence through the encoder twice is its own positivebatch of N = 4 · positives on the diagonal · the other N − 1 columns are free negativestwo dogs run on grassa chef slices onionsthe train is latekids build a sandcastletwo dogs sleep indoorsNLI contradiction of s1(supervised only)BERT encoder[CLS] + MLPpass 1: mask zpass 2: mask z′dropout p = 0.1is the onlyaugmentationrows hᵢcolumns hⱼ⁺h₁⁺h₂⁺h₃⁺h₄⁺h₁⁻h₁0.920.310.120.400.78h₂0.280.890.200.15…h₃0.100.220.940.18…h₄0.370.120.160.90…■ positive ■ in-batch neg ■ hard negrow 1: softmax(cos/τ), τ = 0.05, loss −log p₁₁unsup (4 cols): p₁₁ = 1.0000 → loss 0.00004+ hard neg 0.78: p₁₁ = 0.9426 → loss 0.0591CLIP uses the same matrix across modalities: rows = image embeddings, columns = their captions,N = 32,768 per batch, loss averaged over rows and columns, learnable τ initialised to 0.07
fig.One SimCSE batch. Each sentence is encoded twice with different dropout masks; the diagonal holds the positives and every off-diagonal entry is a free negative. With these example cosines and τ = 0.05, row 1 is already solved (loss 0.00004). Adding the contradiction "two dogs sleep indoors" (cosine 0.78) as a hard negative raises the loss to 0.0591: that column is where the gradient now comes from.

Alignment and uniformity

Wang and Isola (ICML 2020) split what a contrastive loss optimises into two measurable properties of normalised embeddings:

ℓalign=E(x,x+)∥f(x)−f(x+)∥2,ℓuniform=log⁡ Ex,y e−2∥f(x)−f(y)∥2\ell_{\text{align}}=\mathbb{E}_{(x,x^+)}\big\lVert f(x)-f(x^+)\big\rVert^2,\qquad \ell_{\text{uniform}}=\log\,\mathbb{E}_{x,y}\,e^{-2\lVert f(x)-f(y)\rVert^2}
ℓalign\ell_{\text{align}}
positives should land close: lower is better
ℓuniform\ell_{\text{uniform}}
random pairs should spread over the sphere: lower (more negative) is better

The numerator of InfoNCE pulls toward alignment; the denominator pushes toward uniformity. Without negatives the trivial solution is perfect alignment by mapping every input to one point. Pretrained BERT is anisotropic (see Contextual embeddings), i.e. poorly uniform; Gao et al. showed that SimCSE mainly improves uniformity while keeping alignment.

InfoNCE with 7 in-batch negatives · τ = 0.1 · train dropout p = 0.10step 400 of 400 · 8 sentences, linear encoder ℝ³² → ℝ², Adams1s2s3s4s5s6s7s8● f(x) · ● two test views (dropout 0.1)ℓ_uniform = log mean exp(−2‖fᵢ − fⱼ‖²)- - evenly spaced -2.3630−2.4ℓ_align = E‖f(x) − f(x⁺)‖² (test views)0.60step 0400ℓ_uniform -2.022 · ℓ_align 0.096mean cos between sentences -0.077(even spacing: -0.143) · batch loss 0.483negatives spread the cone (uniformity); dropout positives keep each sentence's views together (alignment)
objective
fig.Unsupervised SimCSE on a tiny linear encoder whose 8 outputs live on a circle. Defaults (InfoNCE, τ = 0.1, dropout 0.1): the sentences start in a cone with mean cosine 0.817; at step 20 the cone first tightens (0.893, ℓ_align 0.029), then by step 400 it spreads to −0.077, with ℓ_uniform −0.324 → −2.022 (−2.363 is even spacing) and ℓ_align 0.096. Press positives only: mean cosine 1.000, ℓ_uniform −0.002, a collapse. Set dropout to 0: perfect uniformity (−2.363) but ℓ_align 0.405 on noisy test views.

Reference systems

SimCSE (BERT-base)
average Spearman on 7 STS tasks: 76.3 unsupervised, 81.6 supervised (NLI positives, contradiction hard negatives); τ=0.05\tau=0.05
DPR
Karpukhin et al. (EMNLP 2020): question and passage BERT encoders, in-batch negatives plus one BM25 hard negative per question; see Retrieval-augmented generation (RAG)
CLIP
Radford et al. (2021): 400M image–text pairs, batch 32,768, symmetric InfoNCE over images and captions, learnable τ initialised to 0.07; zero-shot ImageNet 76.2% for ViT-L/14@336px, matching the original supervised ResNet-50
Alignment / uniformity
Wang and Isola (ICML 2020), with t=2t=2 in the Gaussian potential as above

Worked example: why hard negatives matter

One anchor has cosine 0.85 with its positive, 0.40 and 0.30 with two easy in-batch negatives, and 0.80 with a hard negative. The gradient of ℓ\ell with respect to each cosine is (pj−1[j=+])/τ(p_j-\mathbb{1}[j=+])/\tau.

Settingp(positive)p(hard neg)Loss∂ℓ/∂cos(hard)
τ = 0.05, easy negatives only0.9999–0.0001–
τ = 0.05, + hard negative0.73100.26890.31345.378
τ = 0.5, easy negatives only0.5749–0.5536–
τ = 0.5, + hard negative0.37820.34220.97240.684
Softmax weights, loss and gradient on the hard negative's cosine (computed in numpy)

At τ=0.05\tau=0.05 the easy negatives contribute almost nothing: their gradient is 0.0025 and 0.0003 without the hard negative, and the loss is 0.0001. The single hard negative takes a gradient of 5.378, about 3,000 times that of the easy negative at 0.40 (0.0018). At τ=0.5\tau=0.5 the weight is shared: the easy negatives still get 0.31 and 0.25. Low temperature plus hard negatives is what teaches fine distinctions.

python
1import torch, torch.nn.functional as F
2from transformers import AutoTokenizer, AutoModel
3
4tok = AutoTokenizer.from_pretrained("bert-base-uncased")
5enc = AutoModel.from_pretrained("bert-base-uncased") # hidden dropout 0.1
6enc.train() # dropout ON: it is the augmentation
7
8def simcse_loss(sentences, tau=0.05):
9 batch = tok(sentences, padding=True, truncation=True, return_tensors="pt")
10 h1 = enc(**batch).last_hidden_state[:, 0] # [CLS], dropout mask z
11 h2 = enc(**batch).last_hidden_state[:, 0] # same input, new mask z'
12 sim = F.cosine_similarity(h1.unsqueeze(1), h2.unsqueeze(0), dim=-1) / tau # (N, N)
13 labels = torch.arange(len(sentences)) # positives on the diagonal
14 return F.cross_entropy(sim, labels)
15
16loss = simcse_loss(["two dogs run on grass", "a chef slices onions", "the train is late"])
17loss.backward()
Unsupervised SimCSE loss: two forward passes in train mode, cross-entropy on the similarity matrix

Use when

  • +Needs only pairs, not graded similarity labels; unsupervised SimCSE needs only raw sentences
  • +In-batch negatives make every encoded item serve N − 1 other rows at no extra forward pass
  • +Directly optimises the geometry used at inference: cosine search over normalised vectors

Avoid when

  • −Quality depends on the negatives: easy ones give near-zero gradient, false ones push true matches apart
  • −Large batches cost memory; gradient caching or cross-device negatives add engineering
  • −Learns similarity, not truth: negation and number changes remain hard unless mined as hard negatives

Check yourself

In unsupervised SimCSE you accidentally call model.eval() before computing both views. What happens?

Check yourself

Lowering τ from 0.5 to 0.05 mainly changes which negatives drive the gradient. How?

Remember

  • ✓InfoNCE on an N×NN\times N cosine matrix: diagonal = positives, off-diagonal = free in-batch negatives, temperature τ\tau.
  • ✓SimCSE: the positive is the same sentence under a second dropout mask (p = 0.1, τ = 0.05); supervised SimCSE adds NLI entailment positives and contradiction hard negatives.
  • ✓Alignment pulls positives together; uniformity, log⁡Ee−2∥f(x)−f(y)∥2\log\mathbb{E}e^{-2\lVert f(x)-f(y)\rVert^2}, spreads the space. Without negatives the encoder collapses.
  • ✓Easy negatives give almost no gradient at small τ; hard negatives carry the signal.
  • ✓CLIP is the same objective across modalities: images vs captions, batch 32,768, learnable τ.
← back to Deep Learning roadmap