InfoNCE loss

Cross-entropy for picking the one positive out of N candidates by similarity / τ. It lower-bounds mutual information by log N − L, powers CPC, SimCLR, MoCo and CLIP, and its temperature decides how hard it pushes on the closest negatives.

14 min read1091 wordsadvanced0 viewsupdated 2026-10-11

Definition

InfoNCE is the loss L=−log⁡exp⁡(s(q,k+)/τ)∑j=1Nexp⁡(s(q,kj)/τ)L=-\log\dfrac{\exp(s(q,k^+)/\tau)}{\sum_{j=1}^{N}\exp(s(q,k_j)/\tau)} for a query qq, one positive key k+k^+ and N−1N-1 negative keys, where ss is a similarity score and τ\tau a temperature. It is an NN-way softmax cross-entropy whose "correct class" is the positive, and minimising it maximises a lower bound on the mutual information between the views that produced qq and k+k^+.

Pair and triplet losses compare an anchor with one negative at a time, and most negatives give no gradient. InfoNCE compares the positive against all N−1N-1 negatives in one softmax, so every negative in the batch contributes, weighted by how confusable it is. That turned out to be the key ingredient of self-supervised representation learning: take two views of the same thing (two crops of an image, an image and its caption, a context and its future), call them positives, call everything else in the batch negatives, and train the encoder to pick out the positive.

The name comes from noise-contrastive estimation (Gutmann and Hyvärinen, 2010), which learns a density by classifying data against noise samples. Van den Oord, Li and Vinyals introduced InfoNCE in Representation Learning with Contrastive Predictive Coding (CPC, 2018), predicting future latent codes of audio, images, text and RL observations, and proved the mutual-information bound. Wu et al. (2018) used the same loss for instance discrimination with τ=0.07\tau=0.07; MoCo (He et al., 2020) kept a queue of 65,536 negative keys; SimCLR (Chen, Kornblith, Norouzi and Hinton, 2020) called its version NT-Xent (normalised temperature-scaled cross-entropy) and drew negatives from large batches; CLIP (Radford et al., 2021) applied it symmetrically to image–text pairs with a batch of 32,768.

Formula and gradient

L=−log⁡es+/τ∑j=1Nesj/τ=−s+τ+log⁡∑j=1Nesj/τ,sj=q⊤kj∥q∥∥kj∥L=-\log\frac{e^{s_+/\tau}}{\sum_{j=1}^{N}e^{s_j/\tau}}=-\frac{s_+}{\tau}+\log\sum_{j=1}^{N}e^{s_j/\tau},\qquad s_j=\frac{q^\top k_j}{\lVert q\rVert\lVert k_j\rVert}
qq
query (anchor) embedding, e.g. one augmented view or an image
kjk_j
NN candidate keys: one positive k+k^+ (index ++) and N−1N-1 negatives, usually the other items in the batch
sjs_j
similarity score; cosine in SimCLR, MoCo and CLIP, a learned bilinear z⊤Wcz^\top W c in CPC
τ\tau
temperature: divides the scores before the softmax; small τ\tau sharpens it
∂L∂s+=p+−1τ,∂L∂sj=pjτ  (j≠+),pj=esj/τ∑iesi/τ\frac{\partial L}{\partial s_+}=\frac{p_+-1}{\tau},\qquad \frac{\partial L}{\partial s_j}=\frac{p_j}{\tau}\ \ (j\ne+),\qquad p_j=\frac{e^{s_j/\tau}}{\sum_i e^{s_i/\tau}}
pjp_j
the softmax probability the model assigns to candidate j being the positive
pj/τp_j/\tau
push on negative jj: proportional to its softmax weight, so the closest negatives get almost all of it
∑j≠+pj=1−p+\sum_{j\ne+}p_j=1-p_+
the total push on negatives exactly balances the pull on the positive

This is the softmax cross-entropy gradient p−onehotp-\text{onehot} divided by τ\tau. A negative that is far from the query has pj≈0p_j\approx0 and is ignored; a negative almost as similar as the positive takes a large share. InfoNCE is therefore hardness-aware by construction, and the temperature sets how sharply: Wang and Liu (2021) show that smaller τ\tau concentrates the penalty on the hardest negatives.

At a glance

Task
identify the positive among NN candidates: an NN-way classification
Chance level
L=log⁡NL=\log N when all scores are equal
MI bound
I(x;c)≥log⁡N−LI(x;c)\ge\log N-L, so the estimate can never exceed log⁡N\log N
Temperature
typical 0.05 to 0.5; 0.07 in instance discrimination, MoCo and CLIP's initial value
Lower limit
with cosine scores, L≥log⁡(1+(N−1)e−2/τ)L\ge\log\big(1+(N-1)e^{-2/\tau}\big), reached when the positive has s=1s=1 and every negative s=−1s=-1
PyTorch
F.cross_entropy(logits / tau, labels) where row ii of logits holds query ii's scores against all keys
unit circle · drag P or negatives1234567Panchordot size ∝ √p · outlined = hardest negativep = softmax(s / τ), s = cos to anchorjsp∂L/∂sP0.910.52010.770.12820.420.0043−0.340.0004−0.870.0005−1.000.00060.260.00170.870.347∂L/∂s₊ = (p₊ − 1)/τ = −4.80 · ∂L/∂sⱼ = pⱼ/ττ = 0.100 · N = 8 · L = −log p₊ = 0.6543 · log N = 2.0794MI bound I ≥ log N − L = 1.4251 nats (never above log N) · best L at this τ: 0.0000hardest negative takes 72.4% of the push on negatives · top-1 correct
fig.An anchor, a positive P at 25° and N − 1 negatives on the unit circle, scored by cosine. At the start (N = 8, τ = 0.1) the positive has s = 0.91 but negative 7 at −30° has s = 0.87, so p₊ = 0.520, L = 0.6543 against a chance level of log 8 = 2.0794, and the bound gives I ≥ 1.4251 nats. Negative 7 takes 72.4% of the push on negatives while the negatives at s ≤ −0.34 get none. Raise τ to 1: the loss rises to 1.5207, the hardest negative's share falls to 26.9%, and even a perfect layout could not go below L = 0.6665. Lower τ to 0.05 and the hardest share rises to 88.1%. Press 'add a hard negative': negative 1 moves to 10°, ahead of the positive, top-1 becomes wrong and L = 1.3533.

Why it bounds mutual information

Let cc be a context (one view) and xx the matching view, drawn from p(x,c)p(x,c), and let the N−1N-1 negatives be drawn independently from p(x)p(x). The best possible classifier for "which of the NN is the real partner of cc" scores each candidate by the density ratio p(x∣c)/p(x)p(x\mid c)/p(x), and van den Oord et al. show that the optimal score in InfoNCE is proportional to exactly that ratio. Plugging it in gives

LNopt ≥ log⁡N−I(x;c)⟺I(x;c) ≥ log⁡N−LNL_N^{\mathrm{opt}}\ \ge\ \log N - I(x;c)\qquad\Longleftrightarrow\qquad I(x;c)\ \ge\ \log N - L_N
I(x;c)I(x;c)
mutual information between the two views, in nats
log⁡N\log N
the most the bound can certify: log⁡8191=9.01\log 8191=9.01 nats for SimCLR's 4096-image batch (8,190 negatives per positive), log⁡32768=10.40\log 32768=10.40 nats for CLIP
LNL_N
the InfoNCE loss of any critic; the better it classifies, the tighter the bound

Two practical lessons follow. First, more negatives tighten the bound, one reason for the large batches of SimCLR and CLIP and the large queue of MoCo. Second, the bound saturates at log⁡N\log N: Poole et al. (2019) and McAllester and Stratos (2020) point out that it cannot report more mutual information than log⁡N\log N, however much there is. Good representations do not need a tight MI estimate, though: Tschannen et al. (2020) showed that maximising tighter MI bounds can even give worse features, so the success of InfoNCE is better explained by what its gradient does to the embedding geometry.

SimCLR and CLIP: two ways to build the N candidates

SimCLR (NT-Xent). A batch of BB images gives 2B2B augmented views. For view ii with partner jj, the other 2B−22B-2 views are negatives. The scores are cosines of projection-head outputs divided by τ\tau, and the loss is averaged over all 2B2B (anchor, positive) orderings. Chen et al. found that L2 normalisation together with a tuned temperature mattered a great deal for the quality of the learned features, and that larger batches helped, as the bound suggests.

CLIP. A batch of BB (image, text) pairs gives a B×BB\times B cosine-similarity matrix. Row ii is a BB-way classification of which caption belongs to image ii, column ii the reverse; the loss is the mean of the two cross-entropies. The temperature is a learned parameter, initialised to the equivalent of 0.070.07 and clipped so the logits are never scaled by more than 100.

LCLIP=12[CE(S/τ, diag)+CE(S⊤/τ, diag)],Sij=u^i⊤v^jL_{\mathrm{CLIP}}=\tfrac12\Big[\mathrm{CE}\big(S/\tau,\ \mathrm{diag}\big)+\mathrm{CE}\big(S^\top/\tau,\ \mathrm{diag}\big)\Big],\qquad S_{ij}=\hat u_i^\top\hat v_j
u^i,v^j\hat u_i,\hat v_j
L2-normalised image and text embeddings
diag\mathrm{diag}
the correct class of row ii is column ii: matched pairs are on the diagonal
1/τ1/\tau
stored as a learnable log-scale, initialised to log⁡(1/0.07)=2.659\log(1/0.07)=2.659, scale clipped at 100

Worked example: the effect of τ

Take N=4N=4 candidates with cosine scores s=(0.9, 0.5, 0.1, −0.3)s=(0.9,\ 0.5,\ 0.1,\ -0.3), the first being the positive. Chance level is log⁡4=1.3863\log4=1.3863.

τp = softmax(s/τ)L = −log p₊log N − L∂L/∂s₊∂L/∂s₂ (hardest neg)
0.1(0.9817, 0.0180, 0.0003, 0.0000)0.01851.3678−0.18320.1798
0.5(0.5741, 0.2579, 0.1159, 0.0521)0.55500.8313−0.85190.5159
1.0(0.4131, 0.2769, 0.1856, 0.1244)0.88410.5022−0.58690.2769
Softmax, loss, MI bound and gradients for three temperatures (computed in Node)

Reading the table

  1. 1At τ=0.5\tau=0.5: s/τ=(1.8,1.0,0.2,−0.6)s/\tau=(1.8,1.0,0.2,-0.6), log⁡∑es/τ=2.3550\log\sum e^{s/\tau}=2.3550, so L=2.3550−1.8=0.5550L=2.3550-1.8=0.5550.
  2. 2The pull on the positive is (p+−1)/τ=(0.5741−1)/0.5=−0.8519(p_+-1)/\tau=(0.5741-1)/0.5=-0.8519; the pushes pj/τ=(0.5159, 0.2318, 0.1042)p_j/\tau=(0.5159,\,0.2318,\,0.1042) add up to 0.85190.8519.
  3. 3At τ=0.1\tau=0.1 the same scores already separate the positive (p+=0.98p_+=0.98), so the loss and every gradient are small, and 98% of the remaining push lands on the hardest negative (0.17980.1798 of 0.18320.1832).
  4. 4At τ=1\tau=1 the softmax is flat: the scores can only differ by 2, so even a perfect encoder could not push LL below log⁡(1+3e−2)=0.3407\log(1+3e^{-2})=0.3407 here. A too-high temperature caps how confident, and how informative, the loss can be.
python
1import torch, torch.nn.functional as F
2
3def info_nce(q, k, tau=0.1):
4 """q, k: (B, d). Row i of k is the positive for row i of q; the other rows are negatives."""
5 q, k = F.normalize(q, dim=1), F.normalize(k, dim=1)
6 logits = q @ k.T / tau # (B, B) cosine / tau
7 labels = torch.arange(q.size(0)) # positives on the diagonal
8 return F.cross_entropy(logits, labels) # chance level = log(B)
9
10def nt_xent(z1, z2, tau=0.5):
11 """SimCLR: z1, z2 are two augmented views of the same B images; 2B - 2 negatives per anchor."""
12 z = F.normalize(torch.cat([z1, z2]), dim=1) # (2B, d)
13 sim = z @ z.T / tau
14 sim.fill_diagonal_(float("-inf")) # an anchor is never its own candidate
15 B = z1.size(0)
16 labels = torch.cat([torch.arange(B, 2 * B), torch.arange(B)])
17 return F.cross_entropy(sim, labels)
18
19def clip_loss(img, txt, logit_scale):
20 """logit_scale = log(1/tau), a learnable scalar initialised to log(1/0.07)."""
21 img, txt = F.normalize(img, dim=1), F.normalize(txt, dim=1)
22 logits = logit_scale.exp().clamp(max=100) * img @ txt.T
23 labels = torch.arange(img.size(0))
24 return (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) / 2
25
26# The worked example: one query, 4 candidates with cosines (0.9, 0.5, 0.1, -0.3)
27s = torch.tensor([[0.9, 0.5, 0.1, -0.3]])
28print(F.cross_entropy(s / 0.5, torch.tensor([0]))) # 0.5550
InfoNCE / NT-Xent (SimCLR) and the symmetric CLIP loss

Use when

  • +Uses every negative in the batch at once, weighted by difficulty: no explicit mining needed
  • +Simple: one cross-entropy over a similarity matrix, fully vectorised
  • +Principled: a lower bound on mutual information, tighter with more negatives
  • +Scales to web data (CLIP) and to self-supervision with no labels at all (SimCLR, MoCo)

Avoid when

  • −Needs many negatives: large batches, memory queues or momentum encoders
  • −The MI estimate saturates at log⁡N\log N and is a poor estimator of large MI
  • −Sensitive to temperature; too small amplifies false negatives, too large caps the signal
  • −In-batch negatives include false negatives when the data has duplicates or shared classes

Check yourself

A CLIP-style model is trained with batch size 1,024. What is the largest mutual information (in nats) the InfoNCE bound can certify, per direction?

Check yourself

In InfoNCE, which negative receives the largest gradient?

Remember

  • ✓L=−log⁡(es+/τ/∑jesj/τ)L=-\log\big(e^{s_+/\tau}/\sum_j e^{s_j/\tau}\big): cross-entropy for picking the positive among NN candidates (CPC, van den Oord et al., 2018).
  • ✓Gradient (p−onehot)/τ(p-\text{onehot})/\tau: each negative is pushed in proportion to its softmax weight, so the loss is hardness-aware.
  • ✓I(x;c)≥log⁡N−LI(x;c)\ge\log N-L: more negatives tighten the bound, but it can never exceed log⁡N\log N.
  • ✓Temperature controls the focus on hard negatives and the lowest reachable loss; 0.07 to 0.5 is typical, CLIP learns it.
  • ✓SimCLR's NT-Xent uses 2B − 2 in-batch negatives per view; CLIP applies it symmetrically to an image–text similarity matrix.
← back to Deep Learning roadmap