Definition
Contrastive language learning trains a text encoder on pairs that should match (a sentence and a perturbed copy, a paraphrase, a question and its passage, a caption and its image) by making larger than the similarity of to every other item in the batch. The standard objective is InfoNCE with in-batch negatives; the reference methods for language are SimCSE (Gao, Yao and Chen, EMNLP 2021) and, across modalities, CLIP (Radford et al., ICML 2021).
The input is a stream of positive pairs. No labels say how similar two texts are; the supervision is only "these two belong together, the rest of the batch does not". The output is an embedding space in which cosine similarity is meaningful, which is exactly what Sentence embeddings need.
The idea is older than transformers. Siamese networks compared signatures in 1993, and Word2Vec's negative sampling is a contrastive loss over words and contexts. The hard part for sentences is the positive pair. Images have meaning-preserving augmentations (crops, colour jitter); text does not, because deleting or swapping one word can flip the meaning.
InfoNCE with in-batch negatives
- embeddings of the two halves of positive pair i
- the other positives in the batch, reused as negatives at no extra encoding cost
- temperature; small τ makes the softmax sharp, so the loss concentrates on the hardest negatives
Each row is an -way classification: find your own partner among candidates. The batch loss is the mean of , i.e. cross-entropy on the similarity matrix with the diagonal as labels. The mutual-information bound and the role of are derived in InfoNCE loss; the pairwise margin version is Contrastive loss.
SimCSE: dropout is the augmentation
Unsupervised SimCSE feeds the same sentence through BERT twice. Dropout (, already inside the network) samples a different mask and each time, so and differ slightly but mean the same. The other sentences in the batch are the negatives. Training used sentences from English Wikipedia, , and the [CLS] vector with an MLP during training.
In the paper's ablations this beat cropping, word deletion, synonym replacement and masking as augmentations, and beat a next-sentence objective. Turning dropout off, or reusing the same mask for both passes, made results much worse: the two views become identical and the task can be solved without learning anything about meaning.
- supervised SimCSE: an NLI hypothesis entailed by premise i
- a hard negative: the contradiction hypothesis of premise . Close in wording, opposite in meaning
Alignment and uniformity
Wang and Isola (ICML 2020) split what a contrastive loss optimises into two measurable properties of normalised embeddings:
- positives should land close: lower is better
- random pairs should spread over the sphere: lower (more negative) is better
The numerator of InfoNCE pulls toward alignment; the denominator pushes toward uniformity. Without negatives the trivial solution is perfect alignment by mapping every input to one point. Pretrained BERT is anisotropic (see Contextual embeddings), i.e. poorly uniform; Gao et al. showed that SimCSE mainly improves uniformity while keeping alignment.
Reference systems
- SimCSE (BERT-base)
- average Spearman on 7 STS tasks: 76.3 unsupervised, 81.6 supervised (NLI positives, contradiction hard negatives);
- DPR
- Karpukhin et al. (EMNLP 2020): question and passage BERT encoders, in-batch negatives plus one BM25 hard negative per question; see Retrieval-augmented generation (RAG)
- CLIP
- Radford et al. (2021): 400M image–text pairs, batch 32,768, symmetric InfoNCE over images and captions, learnable τ initialised to 0.07; zero-shot ImageNet 76.2% for ViT-L/14@336px, matching the original supervised ResNet-50
- Alignment / uniformity
- Wang and Isola (ICML 2020), with in the Gaussian potential as above
Worked example: why hard negatives matter
One anchor has cosine 0.85 with its positive, 0.40 and 0.30 with two easy in-batch negatives, and 0.80 with a hard negative. The gradient of with respect to each cosine is .
| Setting | p(positive) | p(hard neg) | Loss | ∂ℓ/∂cos(hard) |
|---|---|---|---|---|
| τ = 0.05, easy negatives only | 0.9999 | – | 0.0001 | – |
| τ = 0.05, + hard negative | 0.7310 | 0.2689 | 0.3134 | 5.378 |
| τ = 0.5, easy negatives only | 0.5749 | – | 0.5536 | – |
| τ = 0.5, + hard negative | 0.3782 | 0.3422 | 0.9724 | 0.684 |
At the easy negatives contribute almost nothing: their gradient is 0.0025 and 0.0003 without the hard negative, and the loss is 0.0001. The single hard negative takes a gradient of 5.378, about 3,000 times that of the easy negative at 0.40 (0.0018). At the weight is shared: the easy negatives still get 0.31 and 0.25. Low temperature plus hard negatives is what teaches fine distinctions.
1import torch, torch.nn.functional as F2from transformers import AutoTokenizer, AutoModel34tok = AutoTokenizer.from_pretrained("bert-base-uncased")5enc = AutoModel.from_pretrained("bert-base-uncased") # hidden dropout 0.16enc.train() # dropout ON: it is the augmentation78def simcse_loss(sentences, tau=0.05):9 batch = tok(sentences, padding=True, truncation=True, return_tensors="pt")10 h1 = enc(**batch).last_hidden_state[:, 0] # [CLS], dropout mask z11 h2 = enc(**batch).last_hidden_state[:, 0] # same input, new mask z'12 sim = F.cosine_similarity(h1.unsqueeze(1), h2.unsqueeze(0), dim=-1) / tau # (N, N)13 labels = torch.arange(len(sentences)) # positives on the diagonal14 return F.cross_entropy(sim, labels)1516loss = simcse_loss(["two dogs run on grass", "a chef slices onions", "the train is late"])17loss.backward()Use when
- +Needs only pairs, not graded similarity labels; unsupervised SimCSE needs only raw sentences
- +In-batch negatives make every encoded item serve N − 1 other rows at no extra forward pass
- +Directly optimises the geometry used at inference: cosine search over normalised vectors
Avoid when
- −Quality depends on the negatives: easy ones give near-zero gradient, false ones push true matches apart
- −Large batches cost memory; gradient caching or cross-device negatives add engineering
- −Learns similarity, not truth: negation and number changes remain hard unless mined as hard negatives
Check yourself
In unsupervised SimCSE you accidentally call model.eval() before computing both views. What happens?
Check yourself
Lowering τ from 0.5 to 0.05 mainly changes which negatives drive the gradient. How?
Remember
- ✓InfoNCE on an cosine matrix: diagonal = positives, off-diagonal = free in-batch negatives, temperature .
- ✓SimCSE: the positive is the same sentence under a second dropout mask (p = 0.1, τ = 0.05); supervised SimCSE adds NLI entailment positives and contradiction hard negatives.
- ✓Alignment pulls positives together; uniformity, , spreads the space. Without negatives the encoder collapses.
- ✓Easy negatives give almost no gradient at small τ; hard negatives carry the signal.
- ✓CLIP is the same objective across modalities: images vs captions, batch 32,768, learnable τ.