# Contrastive loss

> A loss on pairs of embeddings: pull similar pairs together with ½D², push dissimilar pairs apart with ½max(0, m − D)². The margin m means pairs already far enough apart cost nothing, so the network learns distances, not labels.

Source: https://www.mlfoundry.in/learn/contrastive-loss  
Section: Deep Learning › Loss Functions  
Level: core · 11 min read · updated 2026-10-11

**Contrastive loss:** (pairwise margin loss) is a loss on a **pair** of inputs and a binary label saying whether they are similar. With embeddings $f(x_1),f(x_2)$ and their distance $D=\lVert f(x_1)-f(x_2)\rVert_2$, a similar pair pays $\tfrac12D^2$ and a dissimilar pair pays $\tfrac12\max(0,\,m-D)^2$. Minimising it makes distance in embedding space mean "how different are these".

Ordinary classification losses need a fixed set of classes. Many problems do not have one: verifying that two signatures or two faces belong to the same person, finding near-duplicate images, or learning a low-dimensional map where neighbours stay neighbours. What you do have is pairs labelled *same* or *different*. Contrastive loss turns that weak supervision into a metric: the network $f$ is trained so that the Euclidean distance between its outputs reflects the labels, and new classes need no retraining.

The idea grew out of **siamese networks**: two copies of one network with shared weights, compared at the output. Bromley et al. (1993) used one for signature verification, and Chopra, Hadsell and LeCun (2005) trained a convolutional siamese net for face verification with an energy-based pair loss. The form used today comes from Hadsell, Chopra and LeCun, *Dimensionality Reduction by Learning an Invariant Mapping* (DrLIM, CVPR 2006). They described it as a system of springs: similar pairs are joined by attract-only springs, dissimilar pairs by repulse-only springs that go slack once the pair is at least $m$ apart.

## Formula and gradient

$$
L(x_1,x_2,y)=y\cdot\tfrac12 D^2+(1-y)\cdot\tfrac12\max(0,\,m-D)^2,\qquad D=\lVert f(x_1)-f(x_2)\rVert_2
$$

- $y$: 1 for a similar (positive) pair, 0 for a dissimilar (negative) pair. DrLIM writes the opposite convention, $Y=0$ for similar
- $D$: Euclidean distance between the two embeddings (not squared)
- $m$: the margin: dissimilar pairs with $D\ge m$ contribute zero loss and zero gradient
- $f$: the shared embedding network (one set of weights, applied to both inputs)

$$
\frac{\partial L}{\partial D}=\begin{cases}D & y=1\\ -(m-D) & y=0,\ D<m\\ 0 & y=0,\ D\ge m\end{cases}\qquad \frac{\partial L}{\partial f(x_1)}=\frac{\partial L}{\partial D}\cdot\frac{f(x_1)-f(x_2)}{D}=-\frac{\partial L}{\partial f(x_2)}
$$

- $\frac{f(x_1)-f(x_2)}{D}$: unit vector along the pair: the force always acts along the line joining the two points
- $D$: pull strength on a similar pair: grows linearly with distance, like a spring
- $m-D$: push strength on a dissimilar pair: largest when the two coincide, zero at the margin

The two halves are equal and opposite on the two embeddings, so each pair moves symmetrically. A similar pair gradient is exactly the MSE gradient between two learned vectors. The dissimilar half is a squared hinge on distance: it switches on only when the pair is closer than $m$. Without that switch, the loss could be lowered forever by pushing negatives to infinity, and without negatives at all the trivial solution $f(x)=\text{const}$ would give zero loss.

**At a glance**

- **Input:** pairs $(x_1,x_2)$ with a binary label similar / dissimilar
- **Output:** an embedding network $f$; similarity is read off as distance $D$
- **Range:** $[0,\infty)$ for similar pairs, $[0,\tfrac12m^2]$ for dissimilar pairs
- **Zero loss when:** similar pairs coincide and dissimilar pairs are at least $m$ apart
- **Hyperparameter:** margin $m$; it only has meaning relative to the scale of $f$ (fix the scale by L2-normalising, or let the loss set it)
- **PyTorch:** no `nn.ContrastiveLoss`; write it in two lines, or use `nn.HingeEmbeddingLoss` for the unsquared variant
- **Successors:** triplet loss (relative distances) and InfoNCE (many negatives at once)

> Figure: Six embeddings, three of class A and three of class B, so 6 similar and 9 dissimilar pairs. The right panel puts every pair on its loss curve. At the start the mean loss over the 15 pairs is 0.9087, almost all of it pull (0.8827), and only 4 of the 9 dissimilar pairs are inside the margin m = 1.5. Press 10 steps (learning rate 0.3): the loss falls to 0.3821 and 7 dissimilar pairs are being pushed. After 100 steps the loss is 0.0000, each class has collapsed to one point and every dissimilar pair sits at exactly D = 1.500: pushing stops at the margin. Reset, set m = 1.0 and run 100 steps: the closest dissimilar pair ends at 1.027, because pairs already beyond the margin are never moved.

## Why the margin, and why it is squared

Think of the dissimilar term as a spring that can only push. Its force is $m-D$ at distance $D<m$ and nothing beyond, so a negative pair is only ever moved **out to** the margin, never further. That keeps the loss bounded below, stops the embedding from inflating its scale to make every negative trivially far, and focuses learning on the confusable pairs. In the figure, most of the 9 negative pairs begin outside the margin and get no gradient at all.

Squaring both terms makes the forces continuous: the pull fades to 0 as a similar pair meets, and the push fades to 0 as a negative reaches the margin, so there is no kink to bounce around. Unsquared variants exist. PyTorch's `nn.HingeEmbeddingLoss` uses $D$ for similar pairs and $\max(0,m-D)$ for dissimilar ones, which gives constant-magnitude forces instead of spring-like ones.

> **Learning a metric, not a classifier:** Nothing in the loss mentions class identities. After training, you can embed a person or product that never appeared in training and compare it to others by distance. That is why contrastive and triplet losses power **verification** (is this the same face?) and **retrieval** (find the nearest items), where the set of classes is open.

## Worked example: one pair, both labels

Take $f(x_1)=(1.0,\,0.5)$, $f(x_2)=(0.2,\,-0.1)$ and $m=1.5$. Then $f(x_1)-f(x_2)=(0.8,\,0.6)$ and $D=1.0$. A second pair with $f(x_2)=(-0.6,\,-0.7)$ has difference $(1.6,\,1.2)$ and $D=2.0$.

Loss and gradient for each case (m = 1.5), checked with the figure's code in Node

| pair | label | L | ∂L/∂D | ∂L/∂f(x₁) |
| --- | --- | --- | --- | --- |
| $D=1.0$ | similar | $\tfrac12(1.0)^2=0.5$ | $1.0$ | $(0.8,\,0.6)$ |
| $D=1.0$ | dissimilar | $\tfrac12(1.5-1.0)^2=0.125$ | $-0.5$ | $(-0.4,\,-0.3)$ |
| $D=2.0$ | similar | $\tfrac12(2.0)^2=2.0$ | $2.0$ | $(1.6,\,1.2)$ |
| $D=2.0$ | dissimilar | $\max(0,-0.5)^2/2=0$ | $0$ | $(0,\,0)$ |

**Reading the dissimilar D = 1.0 row**

1. The pair is closer than the margin by $m-D=0.5$, so the hinge is active and $L=\tfrac12\cdot0.5^2=0.125$.
2. $\partial L/\partial D=-(m-D)=-0.5$: increasing the distance lowers the loss.
3. Multiply by the unit vector $(0.8,0.6)/1.0$: $\partial L/\partial f(x_1)=(-0.4,-0.3)$ and $\partial L/\partial f(x_2)=(0.4,0.3)$.
4. Gradient descent moves $f(x_1)$ along $+(0.4,0.3)$ and $f(x_2)$ along $-(0.4,0.3)$: straight apart. Backprop then carries these vectors into the shared weights from both branches.

```python
import torch, torch.nn as nn, torch.nn.functional as F

def contrastive_loss(z1, z2, y, margin=1.0):
    """z1, z2: (B, d) embeddings; y: (B,) float, 1 = similar, 0 = dissimilar."""
    d = F.pairwise_distance(z1, z2)                 # ||z1 - z2 + eps||_2, eps = 1e-6
    pos = y * d.pow(2)
    neg = (1 - y) * F.relu(margin - d).pow(2)
    return 0.5 * (pos + neg).mean()

z1 = torch.tensor([[1.0, 0.5], [1.0, 0.5]], requires_grad=True)
z2 = torch.tensor([[0.2, -0.1], [0.2, -0.1]])
y = torch.tensor([1.0, 0.0])
loss = contrastive_loss(z1, z2, y, margin=1.5)      # (0.5 + 0.125) / 2 = 0.3125 (up to eps)

# Siamese training: ONE encoder applied to both inputs, so gradients from both branches add up
encoder = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 32))
x1, x2 = torch.randn(64, 784), torch.randn(64, 784)
same = torch.randint(0, 2, (64,)).float()
loss = contrastive_loss(encoder(x1), encoder(x2), same, margin=1.0)
loss.backward()

# Built-in, unsquared: input is the distance, target is +1 (similar) / -1 (dissimilar)
hinge = nn.HingeEmbeddingLoss(margin=1.0)           # x if y = 1, max(0, margin - x) if y = -1
loss2 = hinge(F.pairwise_distance(encoder(x1), encoder(x2)), 2 * same - 1)
```

Contrastive loss in PyTorch (siamese setup), plus the built-in unsquared cousin

> **Negatives that are too easy:** With random pairs, most dissimilar pairs are already beyond the margin and contribute exactly zero, as in the figure where 5 of 9 start outside. Training then sees few useful negatives and slows down. Balance positives and negatives in each batch, and mine **hard negatives** (dissimilar pairs that are currently close). The same issue motivated triplet mining and the many-negative InfoNCE loss.

> **The margin depends on the embedding scale:** $m=1$ means nothing until the scale of $f$ is pinned down. On unnormalised embeddings the network can trade scale against the margin; on L2-normalised embeddings all distances lie in $[0,2]$, so $m$ must be chosen inside that range. Also note the convention flip: DrLIM uses $Y=0$ for **similar** pairs, many codebases use $y=1$. Check which before reusing a margin from a paper.

> **Where it sits among metric losses:** Contrastive loss constrains **absolute** distances: positives near 0, negatives beyond $m$. Triplet loss only constrains **relative** ones, $d(a,p)+m<d(a,n)$, which tolerates classes of different spread. InfoNCE replaces the single negative with a softmax over many. Self-supervised methods such as SimCLR are often called "contrastive learning", but they use InfoNCE, not this pairwise loss.

**Strengths**

- Needs only same / different labels, no fixed class list: works for verification and open-set retrieval
- Simple, smooth, two-line implementation with spring-like, interpretable forces
- The margin bounds the push, so the embedding cannot win by inflating its scale
- Each pair is independent: easy to batch and to combine with other losses

**Limitations**

- Absolute targets (0 and ≥ m) for every pair, even for classes that are naturally spread out
- Most random negatives give zero gradient; needs balanced sampling or hard-negative mining
- Margin must match the embedding scale; collapse of a class to one point is encouraged
- Looks at one negative at a time, so it learns more slowly than multi-negative losses

**Question:** With margin m = 1.5, a dissimilar pair has D = 2.0. What does it contribute to the gradient?

- A. Nothing: the pair is beyond the margin, so its loss and gradient are 0
- B. A small push apart proportional to D − m
- C. A pull together, because D > m
- D. A push apart of strength m

**Answer:** A. Nothing: the pair is beyond the margin, so its loss and gradient are 0 — The dissimilar term is $\tfrac12\max(0,m-D)^2$. For $D\ge m$ the hinge is off, so $L=0$ and $\partial L/\partial D=0$. Negative pairs are only pushed out to the margin, never further.

**Question:** Why can't you train an embedding with only the similar-pair term ½D²?

- A. Every input mapped to the same point gives zero loss, so the network collapses
- B. Its gradient is not defined at D = 0
- C. It is not differentiable with respect to the weights
- D. It needs a margin to be convex

**Answer:** A. Every input mapped to the same point gives zero loss, so the network collapses — If $f(x)=c$ for all $x$, every distance is 0 and the pull term is minimised exactly. The dissimilar term is what forbids that solution; the margin then stops it from pushing negatives to infinity instead.

**Key points**

- $L=y\,\tfrac12D^2+(1-y)\,\tfrac12\max(0,m-D)^2$ on pairs, with $D$ the distance between shared-network embeddings (Hadsell, Chopra and LeCun, 2006).
- Gradient on each embedding is $\partial L/\partial D$ times the unit vector along the pair: a spring pull $D$, or a push $m-D$ that stops at the margin.
- Dissimilar pairs beyond the margin give exactly zero gradient, so sampling and hard-negative mining matter.
- The margin is meaningful only relative to the embedding scale; check the label convention (DrLIM uses Y = 0 for similar).
- It learns a metric, which is what verification and retrieval with unseen classes need.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
