Definition
InfoNCE is the loss for a query , one positive key and negative keys, where is a similarity score and a temperature. It is an -way softmax cross-entropy whose "correct class" is the positive, and minimising it maximises a lower bound on the mutual information between the views that produced and .
Pair and triplet losses compare an anchor with one negative at a time, and most negatives give no gradient. InfoNCE compares the positive against all negatives in one softmax, so every negative in the batch contributes, weighted by how confusable it is. That turned out to be the key ingredient of self-supervised representation learning: take two views of the same thing (two crops of an image, an image and its caption, a context and its future), call them positives, call everything else in the batch negatives, and train the encoder to pick out the positive.
The name comes from noise-contrastive estimation (Gutmann and Hyvärinen, 2010), which learns a density by classifying data against noise samples. Van den Oord, Li and Vinyals introduced InfoNCE in Representation Learning with Contrastive Predictive Coding (CPC, 2018), predicting future latent codes of audio, images, text and RL observations, and proved the mutual-information bound. Wu et al. (2018) used the same loss for instance discrimination with ; MoCo (He et al., 2020) kept a queue of 65,536 negative keys; SimCLR (Chen, Kornblith, Norouzi and Hinton, 2020) called its version NT-Xent (normalised temperature-scaled cross-entropy) and drew negatives from large batches; CLIP (Radford et al., 2021) applied it symmetrically to image–text pairs with a batch of 32,768.
Formula and gradient
- query (anchor) embedding, e.g. one augmented view or an image
- candidate keys: one positive (index ) and negatives, usually the other items in the batch
- similarity score; cosine in SimCLR, MoCo and CLIP, a learned bilinear in CPC
- temperature: divides the scores before the softmax; small sharpens it
- the softmax probability the model assigns to candidate j being the positive
- push on negative : proportional to its softmax weight, so the closest negatives get almost all of it
- the total push on negatives exactly balances the pull on the positive
This is the softmax cross-entropy gradient divided by . A negative that is far from the query has and is ignored; a negative almost as similar as the positive takes a large share. InfoNCE is therefore hardness-aware by construction, and the temperature sets how sharply: Wang and Liu (2021) show that smaller concentrates the penalty on the hardest negatives.
At a glance
- Task
- identify the positive among candidates: an -way classification
- Chance level
- when all scores are equal
- MI bound
- , so the estimate can never exceed
- Temperature
- typical 0.05 to 0.5; 0.07 in instance discrimination, MoCo and CLIP's initial value
- Lower limit
- with cosine scores, , reached when the positive has and every negative
- PyTorch
F.cross_entropy(logits / tau, labels)where row oflogitsholds query 's scores against all keys
Why it bounds mutual information
Let be a context (one view) and the matching view, drawn from , and let the negatives be drawn independently from . The best possible classifier for "which of the is the real partner of " scores each candidate by the density ratio , and van den Oord et al. show that the optimal score in InfoNCE is proportional to exactly that ratio. Plugging it in gives
- mutual information between the two views, in nats
- the most the bound can certify: nats for SimCLR's 4096-image batch (8,190 negatives per positive), nats for CLIP
- the InfoNCE loss of any critic; the better it classifies, the tighter the bound
Two practical lessons follow. First, more negatives tighten the bound, one reason for the large batches of SimCLR and CLIP and the large queue of MoCo. Second, the bound saturates at : Poole et al. (2019) and McAllester and Stratos (2020) point out that it cannot report more mutual information than , however much there is. Good representations do not need a tight MI estimate, though: Tschannen et al. (2020) showed that maximising tighter MI bounds can even give worse features, so the success of InfoNCE is better explained by what its gradient does to the embedding geometry.
SimCLR and CLIP: two ways to build the N candidates
SimCLR (NT-Xent). A batch of images gives augmented views. For view with partner , the other views are negatives. The scores are cosines of projection-head outputs divided by , and the loss is averaged over all (anchor, positive) orderings. Chen et al. found that L2 normalisation together with a tuned temperature mattered a great deal for the quality of the learned features, and that larger batches helped, as the bound suggests.
CLIP. A batch of (image, text) pairs gives a cosine-similarity matrix. Row is a -way classification of which caption belongs to image , column the reverse; the loss is the mean of the two cross-entropies. The temperature is a learned parameter, initialised to the equivalent of and clipped so the logits are never scaled by more than 100.
- L2-normalised image and text embeddings
- the correct class of row is column : matched pairs are on the diagonal
- stored as a learnable log-scale, initialised to , scale clipped at 100
Worked example: the effect of τ
Take candidates with cosine scores , the first being the positive. Chance level is .
| τ | p = softmax(s/τ) | L = −log p₊ | log N − L | ∂L/∂s₊ | ∂L/∂s₂ (hardest neg) |
|---|---|---|---|---|---|
| 0.1 | (0.9817, 0.0180, 0.0003, 0.0000) | 0.0185 | 1.3678 | −0.1832 | 0.1798 |
| 0.5 | (0.5741, 0.2579, 0.1159, 0.0521) | 0.5550 | 0.8313 | −0.8519 | 0.5159 |
| 1.0 | (0.4131, 0.2769, 0.1856, 0.1244) | 0.8841 | 0.5022 | −0.5869 | 0.2769 |
Reading the table
- 1At : , , so .
- 2The pull on the positive is ; the pushes add up to .
- 3At the same scores already separate the positive (), so the loss and every gradient are small, and 98% of the remaining push lands on the hardest negative ( of ).
- 4At the softmax is flat: the scores can only differ by 2, so even a perfect encoder could not push below here. A too-high temperature caps how confident, and how informative, the loss can be.
1import torch, torch.nn.functional as F23def info_nce(q, k, tau=0.1):4 """q, k: (B, d). Row i of k is the positive for row i of q; the other rows are negatives."""5 q, k = F.normalize(q, dim=1), F.normalize(k, dim=1)6 logits = q @ k.T / tau # (B, B) cosine / tau7 labels = torch.arange(q.size(0)) # positives on the diagonal8 return F.cross_entropy(logits, labels) # chance level = log(B)910def nt_xent(z1, z2, tau=0.5):11 """SimCLR: z1, z2 are two augmented views of the same B images; 2B - 2 negatives per anchor."""12 z = F.normalize(torch.cat([z1, z2]), dim=1) # (2B, d)13 sim = z @ z.T / tau14 sim.fill_diagonal_(float("-inf")) # an anchor is never its own candidate15 B = z1.size(0)16 labels = torch.cat([torch.arange(B, 2 * B), torch.arange(B)])17 return F.cross_entropy(sim, labels)1819def clip_loss(img, txt, logit_scale):20 """logit_scale = log(1/tau), a learnable scalar initialised to log(1/0.07)."""21 img, txt = F.normalize(img, dim=1), F.normalize(txt, dim=1)22 logits = logit_scale.exp().clamp(max=100) * img @ txt.T23 labels = torch.arange(img.size(0))24 return (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) / 22526# The worked example: one query, 4 candidates with cosines (0.9, 0.5, 0.1, -0.3)27s = torch.tensor([[0.9, 0.5, 0.1, -0.3]])28print(F.cross_entropy(s / 0.5, torch.tensor([0]))) # 0.5550Use when
- +Uses every negative in the batch at once, weighted by difficulty: no explicit mining needed
- +Simple: one cross-entropy over a similarity matrix, fully vectorised
- +Principled: a lower bound on mutual information, tighter with more negatives
- +Scales to web data (CLIP) and to self-supervision with no labels at all (SimCLR, MoCo)
Avoid when
- −Needs many negatives: large batches, memory queues or momentum encoders
- −The MI estimate saturates at and is a poor estimator of large MI
- −Sensitive to temperature; too small amplifies false negatives, too large caps the signal
- −In-batch negatives include false negatives when the data has duplicates or shared classes
Check yourself
A CLIP-style model is trained with batch size 1,024. What is the largest mutual information (in nats) the InfoNCE bound can certify, per direction?
Check yourself
In InfoNCE, which negative receives the largest gradient?
Remember
- ✓: cross-entropy for picking the positive among candidates (CPC, van den Oord et al., 2018).
- ✓Gradient : each negative is pushed in proportion to its softmax weight, so the loss is hardness-aware.
- ✓: more negatives tighten the bound, but it can never exceed .
- ✓Temperature controls the focus on hard negatives and the lowest reachable loss; 0.07 to 0.5 is typical, CLIP learns it.
- ✓SimCLR's NT-Xent uses 2B − 2 in-batch negatives per view; CLIP applies it symmetrically to an image–text similarity matrix.