# InfoNCE loss

> Cross-entropy for picking the one positive out of N candidates by similarity / τ. It lower-bounds mutual information by log N − L, powers CPC, SimCLR, MoCo and CLIP, and its temperature decides how hard it pushes on the closest negatives.

Source: https://www.mlfoundry.in/learn/infonce-loss  
Section: Deep Learning › Loss Functions  
Level: advanced · 14 min read · updated 2026-10-11

**InfoNCE:** is the loss $L=-\log\dfrac{\exp(s(q,k^+)/\tau)}{\sum_{j=1}^{N}\exp(s(q,k_j)/\tau)}$ for a query $q$, one positive key $k^+$ and $N-1$ negative keys, where $s$ is a similarity score and $\tau$ a temperature. It is an $N$-way softmax cross-entropy whose "correct class" is the positive, and minimising it maximises a lower bound on the mutual information between the views that produced $q$ and $k^+$.

Pair and triplet losses compare an anchor with one negative at a time, and most negatives give no gradient. InfoNCE compares the positive against **all** $N-1$ negatives in one softmax, so every negative in the batch contributes, weighted by how confusable it is. That turned out to be the key ingredient of self-supervised representation learning: take two views of the same thing (two crops of an image, an image and its caption, a context and its future), call them positives, call everything else in the batch negatives, and train the encoder to pick out the positive.

The name comes from **noise-contrastive estimation** (Gutmann and Hyvärinen, 2010), which learns a density by classifying data against noise samples. Van den Oord, Li and Vinyals introduced InfoNCE in *Representation Learning with Contrastive Predictive Coding* (CPC, 2018), predicting future latent codes of audio, images, text and RL observations, and proved the mutual-information bound. Wu et al. (2018) used the same loss for instance discrimination with $\tau=0.07$; MoCo (He et al., 2020) kept a queue of 65,536 negative keys; SimCLR (Chen, Kornblith, Norouzi and Hinton, 2020) called its version **NT-Xent** (normalised temperature-scaled cross-entropy) and drew negatives from large batches; CLIP (Radford et al., 2021) applied it symmetrically to image–text pairs with a batch of 32,768.

## Formula and gradient

$$
L=-\log\frac{e^{s_+/\tau}}{\sum_{j=1}^{N}e^{s_j/\tau}}=-\frac{s_+}{\tau}+\log\sum_{j=1}^{N}e^{s_j/\tau},\qquad s_j=\frac{q^\top k_j}{\lVert q\rVert\lVert k_j\rVert}
$$

- $q$: query (anchor) embedding, e.g. one augmented view or an image
- $k_j$: $N$ candidate keys: one positive $k^+$ (index $+$) and $N-1$ negatives, usually the other items in the batch
- $s_j$: similarity score; cosine in SimCLR, MoCo and CLIP, a learned bilinear $z^\top W c$ in CPC
- $\tau$: temperature: divides the scores before the softmax; small $\tau$ sharpens it

$$
\frac{\partial L}{\partial s_+}=\frac{p_+-1}{\tau},\qquad \frac{\partial L}{\partial s_j}=\frac{p_j}{\tau}\ \ (j\ne+),\qquad p_j=\frac{e^{s_j/\tau}}{\sum_i e^{s_i/\tau}}
$$

- $p_j$: the softmax probability the model assigns to candidate j being the positive
- $p_j/\tau$: push on negative $j$: proportional to its softmax weight, so the closest negatives get almost all of it
- $\sum_{j\ne+}p_j=1-p_+$: the total push on negatives exactly balances the pull on the positive

This is the softmax cross-entropy gradient $p-\text{onehot}$ divided by $\tau$. A negative that is far from the query has $p_j\approx0$ and is ignored; a negative almost as similar as the positive takes a large share. InfoNCE is therefore **hardness-aware by construction**, and the temperature sets how sharply: Wang and Liu (2021) show that smaller $\tau$ concentrates the penalty on the hardest negatives.

**At a glance**

- **Task:** identify the positive among $N$ candidates: an $N$-way classification
- **Chance level:** $L=\log N$ when all scores are equal
- **MI bound:** $I(x;c)\ge\log N-L$, so the estimate can never exceed $\log N$
- **Temperature:** typical 0.05 to 0.5; 0.07 in instance discrimination, MoCo and CLIP's initial value
- **Lower limit:** with cosine scores, $L\ge\log\big(1+(N-1)e^{-2/\tau}\big)$, reached when the positive has $s=1$ and every negative $s=-1$
- **PyTorch:** `F.cross_entropy(logits / tau, labels)` where row $i$ of `logits` holds query $i$'s scores against all keys

> Figure: An anchor, a positive P at 25° and N − 1 negatives on the unit circle, scored by cosine. At the start (N = 8, τ = 0.1) the positive has s = 0.91 but negative 7 at −30° has s = 0.87, so p₊ = 0.520, L = 0.6543 against a chance level of log 8 = 2.0794, and the bound gives I ≥ 1.4251 nats. Negative 7 takes 72.4% of the push on negatives while the negatives at s ≤ −0.34 get none. Raise τ to 1: the loss rises to 1.5207, the hardest negative's share falls to 26.9%, and even a perfect layout could not go below L = 0.6665. Lower τ to 0.05 and the hardest share rises to 88.1%. Press 'add a hard negative': negative 1 moves to 10°, ahead of the positive, top-1 becomes wrong and L = 1.3533.

## Why it bounds mutual information

Let $c$ be a context (one view) and $x$ the matching view, drawn from $p(x,c)$, and let the $N-1$ negatives be drawn independently from $p(x)$. The best possible classifier for "which of the $N$ is the real partner of $c$" scores each candidate by the density ratio $p(x\mid c)/p(x)$, and van den Oord et al. show that the optimal score in InfoNCE is proportional to exactly that ratio. Plugging it in gives

$$
L_N^{\mathrm{opt}}\ \ge\ \log N - I(x;c)\qquad\Longleftrightarrow\qquad I(x;c)\ \ge\ \log N - L_N
$$

- $I(x;c)$: mutual information between the two views, in nats
- $\log N$: the most the bound can certify: $\log 8191=9.01$ nats for SimCLR's 4096-image batch (8,190 negatives per positive), $\log 32768=10.40$ nats for CLIP
- $L_N$: the InfoNCE loss of any critic; the better it classifies, the tighter the bound

Two practical lessons follow. First, more negatives tighten the bound, one reason for the large batches of SimCLR and CLIP and the large queue of MoCo. Second, the bound **saturates** at $\log N$: Poole et al. (2019) and McAllester and Stratos (2020) point out that it cannot report more mutual information than $\log N$, however much there is. Good representations do not need a tight MI estimate, though: Tschannen et al. (2020) showed that maximising tighter MI bounds can even give worse features, so the success of InfoNCE is better explained by what its gradient does to the embedding geometry.

> **Alignment and uniformity:** Wang and Isola (2020) decompose the normalised InfoNCE loss for large $N$ into two terms: **alignment**, which pulls positive pairs together on the sphere, and **uniformity**, a log-sum-exp over all pairs that spreads embeddings evenly over the sphere. The negatives are what prevent collapse: without the denominator, mapping everything to one point would be optimal.

## SimCLR and CLIP: two ways to build the N candidates

**SimCLR (NT-Xent).** A batch of $B$ images gives $2B$ augmented views. For view $i$ with partner $j$, the other $2B-2$ views are negatives. The scores are cosines of projection-head outputs divided by $\tau$, and the loss is averaged over all $2B$ (anchor, positive) orderings. Chen et al. found that L2 normalisation together with a tuned temperature mattered a great deal for the quality of the learned features, and that larger batches helped, as the bound suggests.

**CLIP.** A batch of $B$ (image, text) pairs gives a $B\times B$ cosine-similarity matrix. Row $i$ is a $B$-way classification of which caption belongs to image $i$, column $i$ the reverse; the loss is the mean of the two cross-entropies. The temperature is a learned parameter, initialised to the equivalent of $0.07$ and clipped so the logits are never scaled by more than 100.

$$
L_{\mathrm{CLIP}}=\tfrac12\Big[\mathrm{CE}\big(S/\tau,\ \mathrm{diag}\big)+\mathrm{CE}\big(S^\top/\tau,\ \mathrm{diag}\big)\Big],\qquad S_{ij}=\hat u_i^\top\hat v_j
$$

- $\hat u_i,\hat v_j$: L2-normalised image and text embeddings
- $\mathrm{diag}$: the correct class of row $i$ is column $i$: matched pairs are on the diagonal
- $1/\tau$: stored as a learnable log-scale, initialised to $\log(1/0.07)=2.659$, scale clipped at 100

## Worked example: the effect of τ

Take $N=4$ candidates with cosine scores $s=(0.9,\ 0.5,\ 0.1,\ -0.3)$, the first being the positive. Chance level is $\log4=1.3863$.

Softmax, loss, MI bound and gradients for three temperatures (computed in Node)

| τ | p = softmax(s/τ) | L = −log p₊ | log N − L | ∂L/∂s₊ | ∂L/∂s₂ (hardest neg) |
| --- | --- | --- | --- | --- | --- |
| 0.1 | (0.9817, 0.0180, 0.0003, 0.0000) | 0.0185 | 1.3678 | −0.1832 | 0.1798 |
| 0.5 | (0.5741, 0.2579, 0.1159, 0.0521) | 0.5550 | 0.8313 | −0.8519 | 0.5159 |
| 1.0 | (0.4131, 0.2769, 0.1856, 0.1244) | 0.8841 | 0.5022 | −0.5869 | 0.2769 |

**Reading the table**

1. At $\tau=0.5$: $s/\tau=(1.8,1.0,0.2,-0.6)$, $\log\sum e^{s/\tau}=2.3550$, so $L=2.3550-1.8=0.5550$.
2. The pull on the positive is $(p_+-1)/\tau=(0.5741-1)/0.5=-0.8519$; the pushes $p_j/\tau=(0.5159,\,0.2318,\,0.1042)$ add up to $0.8519$.
3. At $\tau=0.1$ the same scores already separate the positive ($p_+=0.98$), so the loss and every gradient are small, and 98% of the remaining push lands on the hardest negative ($0.1798$ of $0.1832$).
4. At $\tau=1$ the softmax is flat: the scores can only differ by 2, so even a perfect encoder could not push $L$ below $\log(1+3e^{-2})=0.3407$ here. A too-high temperature caps how confident, and how informative, the loss can be.

```python
import torch, torch.nn.functional as F

def info_nce(q, k, tau=0.1):
    """q, k: (B, d). Row i of k is the positive for row i of q; the other rows are negatives."""
    q, k = F.normalize(q, dim=1), F.normalize(k, dim=1)
    logits = q @ k.T / tau                          # (B, B) cosine / tau
    labels = torch.arange(q.size(0))                # positives on the diagonal
    return F.cross_entropy(logits, labels)          # chance level = log(B)

def nt_xent(z1, z2, tau=0.5):
    """SimCLR: z1, z2 are two augmented views of the same B images; 2B - 2 negatives per anchor."""
    z = F.normalize(torch.cat([z1, z2]), dim=1)     # (2B, d)
    sim = z @ z.T / tau
    sim.fill_diagonal_(float("-inf"))               # an anchor is never its own candidate
    B = z1.size(0)
    labels = torch.cat([torch.arange(B, 2 * B), torch.arange(B)])
    return F.cross_entropy(sim, labels)

def clip_loss(img, txt, logit_scale):
    """logit_scale = log(1/tau), a learnable scalar initialised to log(1/0.07)."""
    img, txt = F.normalize(img, dim=1), F.normalize(txt, dim=1)
    logits = logit_scale.exp().clamp(max=100) * img @ txt.T
    labels = torch.arange(img.size(0))
    return (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) / 2

# The worked example: one query, 4 candidates with cosines (0.9, 0.5, 0.1, -0.3)
s = torch.tensor([[0.9, 0.5, 0.1, -0.3]])
print(F.cross_entropy(s / 0.5, torch.tensor([0])))   # 0.5550
```

InfoNCE / NT-Xent (SimCLR) and the symmetric CLIP loss

> **False negatives:** "Everything else in the batch is a negative" is only approximately true. Two different photos of dogs, or two captions that both fit an image, are treated as negatives and pushed apart, and the softmax gives them a **large** share of the gradient precisely because they are similar. With small $\tau$ and large batches this gets worse. Remedies include debiased contrastive losses, supervised contrastive loss (using labels to mark same-class items as positives), or simply not using too small a temperature.

> **Temperature is not a detail:** Too large a $\tau$ flattens the softmax: in the figure at $\tau=1$, eight candidates on the circle cannot get $L$ below 0.6665, and the hardest negative gets only 26.9% of the push. Too small a $\tau$ makes the loss focus on a handful of hardest (possibly false) negatives and can make training unstable. Tune it, or learn it as CLIP does, with a clamp.

> **Relatives:** With $N=2$ and a hinge instead of the softmax, InfoNCE reduces to a triplet-style comparison. The N-pair loss (Sohn, 2016) is InfoNCE without temperature on unnormalised dot products. Supervised contrastive loss (Khosla et al., 2020) allows several positives per anchor. The cross-entropy in classification is InfoNCE where the keys are learned class prototypes.

**Strengths**

- Uses every negative in the batch at once, weighted by difficulty: no explicit mining needed
- Simple: one cross-entropy over a similarity matrix, fully vectorised
- Principled: a lower bound on mutual information, tighter with more negatives
- Scales to web data (CLIP) and to self-supervision with no labels at all (SimCLR, MoCo)

**Limitations**

- Needs many negatives: large batches, memory queues or momentum encoders
- The MI estimate saturates at $\log N$ and is a poor estimator of large MI
- Sensitive to temperature; too small amplifies false negatives, too large caps the signal
- In-batch negatives include false negatives when the data has duplicates or shared classes

**Question:** A CLIP-style model is trained with batch size 1,024. What is the largest mutual information (in nats) the InfoNCE bound can certify, per direction?

- A. log 1024 ≈ 6.93
- B. 1024
- C. 1024² ≈ 1.05 million
- D. It is unbounded if the model is perfect

**Answer:** A. log 1024 ≈ 6.93 — Each row is a 1,024-way classification, so $I\ge\log N-L\le\log1024=6.93$ nats even at $L=0$. The bound saturates at $\log N$, however much information the two modalities share.

**Question:** In InfoNCE, which negative receives the largest gradient?

- A. The one with the highest similarity to the query
- B. All negatives get the same gradient
- C. The one with the lowest similarity
- D. The one closest to the positive key

**Answer:** A. The one with the highest similarity to the query — $\partial L/\partial s_j=p_j/\tau$ and $p_j$ grows with $s_j$, so the most similar (hardest) negative gets the biggest push. Lower $\tau$ concentrates it further: 72.4% of the push on the hardest negative at $\tau=0.1$ in the figure, 88.1% at $\tau=0.05$.

**Key points**

- $L=-\log\big(e^{s_+/\tau}/\sum_j e^{s_j/\tau}\big)$: cross-entropy for picking the positive among $N$ candidates (CPC, van den Oord et al., 2018).
- Gradient $(p-\text{onehot})/\tau$: each negative is pushed in proportion to its softmax weight, so the loss is hardness-aware.
- $I(x;c)\ge\log N-L$: more negatives tighten the bound, but it can never exceed $\log N$.
- Temperature controls the focus on hard negatives and the lowest reachable loss; 0.07 to 0.5 is typical, CLIP learns it.
- SimCLR's NT-Xent uses 2B − 2 in-batch negatives per view; CLIP applies it symmetrically to an image–text similarity matrix.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
