Contrastive loss

A loss on pairs of embeddings: pull similar pairs together with ½D², push dissimilar pairs apart with ½max(0, m − D)². The margin m means pairs already far enough apart cost nothing, so the network learns distances, not labels.

11 min read851 wordscore0 viewsupdated 2026-10-11

Definition

Contrastive loss (pairwise margin loss) is a loss on a pair of inputs and a binary label saying whether they are similar. With embeddings f(x1),f(x2)f(x_1),f(x_2) and their distance D=∥f(x1)−f(x2)∥2D=\lVert f(x_1)-f(x_2)\rVert_2, a similar pair pays 12D2\tfrac12D^2 and a dissimilar pair pays 12max⁡(0, m−D)2\tfrac12\max(0,\,m-D)^2. Minimising it makes distance in embedding space mean "how different are these".

Ordinary classification losses need a fixed set of classes. Many problems do not have one: verifying that two signatures or two faces belong to the same person, finding near-duplicate images, or learning a low-dimensional map where neighbours stay neighbours. What you do have is pairs labelled same or different. Contrastive loss turns that weak supervision into a metric: the network ff is trained so that the Euclidean distance between its outputs reflects the labels, and new classes need no retraining.

The idea grew out of siamese networks: two copies of one network with shared weights, compared at the output. Bromley et al. (1993) used one for signature verification, and Chopra, Hadsell and LeCun (2005) trained a convolutional siamese net for face verification with an energy-based pair loss. The form used today comes from Hadsell, Chopra and LeCun, Dimensionality Reduction by Learning an Invariant Mapping (DrLIM, CVPR 2006). They described it as a system of springs: similar pairs are joined by attract-only springs, dissimilar pairs by repulse-only springs that go slack once the pair is at least mm apart.

Formula and gradient

L(x1,x2,y)=y⋅12D2+(1−y)⋅12max⁡(0, m−D)2,D=∥f(x1)−f(x2)∥2L(x_1,x_2,y)=y\cdot\tfrac12 D^2+(1-y)\cdot\tfrac12\max(0,\,m-D)^2,\qquad D=\lVert f(x_1)-f(x_2)\rVert_2
yy
1 for a similar (positive) pair, 0 for a dissimilar (negative) pair. DrLIM writes the opposite convention, Y=0Y=0 for similar
DD
Euclidean distance between the two embeddings (not squared)
mm
the margin: dissimilar pairs with D≥mD\ge m contribute zero loss and zero gradient
ff
the shared embedding network (one set of weights, applied to both inputs)
∂L∂D={Dy=1−(m−D)y=0, D<m0y=0, D≥m∂L∂f(x1)=∂L∂D⋅f(x1)−f(x2)D=−∂L∂f(x2)\frac{\partial L}{\partial D}=\begin{cases}D & y=1\\ -(m-D) & y=0,\ D<m\\ 0 & y=0,\ D\ge m\end{cases}\qquad \frac{\partial L}{\partial f(x_1)}=\frac{\partial L}{\partial D}\cdot\frac{f(x_1)-f(x_2)}{D}=-\frac{\partial L}{\partial f(x_2)}
f(x1)−f(x2)D\frac{f(x_1)-f(x_2)}{D}
unit vector along the pair: the force always acts along the line joining the two points
DD
pull strength on a similar pair: grows linearly with distance, like a spring
m−Dm-D
push strength on a dissimilar pair: largest when the two coincide, zero at the margin

The two halves are equal and opposite on the two embeddings, so each pair moves symmetrically. A similar pair gradient is exactly the MSE gradient between two learned vectors. The dissimilar half is a squared hinge on distance: it switches on only when the pair is closer than mm. Without that switch, the loss could be lowered forever by pushing negatives to infinity, and without negatives at all the trivial solution f(x)=constf(x)=\text{const} would give zero loss.

At a glance

Input
pairs (x1,x2)(x_1,x_2) with a binary label similar / dissimilar
Output
an embedding network ff; similarity is read off as distance DD
Range
[0,∞)[0,\infty) for similar pairs, [0,12m2][0,\tfrac12m^2] for dissimilar pairs
Zero loss when
similar pairs coincide and dissimilar pairs are at least mm apart
Hyperparameter
margin mm; it only has meaning relative to the scale of ff (fix the scale by L2-normalising, or let the loss set it)
PyTorch
no nn.ContrastiveLoss; write it in two lines, or use nn.HingeEmbeddingLoss for the unsquared variant
Successors
triplet loss (relative distances) and InfoNCE (many negatives at once)
embeddings (drag) · arrows = −∂L/∂xAAABBBper-pair loss vs distance D01230123m = 1.50— similar ½D²— dissimilar ½max(0, m−D)²dots = pairs · rose right of m: no forcemean L over 15 pairs = 0.9087 = pull 0.8827 + push 0.0260 · step 0dissimilar pairs inside the margin (dashed, pushed): 4 of 9
gradient descent (lr 0.3)
fig.Six embeddings, three of class A and three of class B, so 6 similar and 9 dissimilar pairs. The right panel puts every pair on its loss curve. At the start the mean loss over the 15 pairs is 0.9087, almost all of it pull (0.8827), and only 4 of the 9 dissimilar pairs are inside the margin m = 1.5. Press 10 steps (learning rate 0.3): the loss falls to 0.3821 and 7 dissimilar pairs are being pushed. After 100 steps the loss is 0.0000, each class has collapsed to one point and every dissimilar pair sits at exactly D = 1.500: pushing stops at the margin. Reset, set m = 1.0 and run 100 steps: the closest dissimilar pair ends at 1.027, because pairs already beyond the margin are never moved.

Why the margin, and why it is squared

Think of the dissimilar term as a spring that can only push. Its force is m−Dm-D at distance D<mD<m and nothing beyond, so a negative pair is only ever moved out to the margin, never further. That keeps the loss bounded below, stops the embedding from inflating its scale to make every negative trivially far, and focuses learning on the confusable pairs. In the figure, most of the 9 negative pairs begin outside the margin and get no gradient at all.

Squaring both terms makes the forces continuous: the pull fades to 0 as a similar pair meets, and the push fades to 0 as a negative reaches the margin, so there is no kink to bounce around. Unsquared variants exist. PyTorch's nn.HingeEmbeddingLoss uses DD for similar pairs and max⁡(0,m−D)\max(0,m-D) for dissimilar ones, which gives constant-magnitude forces instead of spring-like ones.

Worked example: one pair, both labels

Take f(x1)=(1.0, 0.5)f(x_1)=(1.0,\,0.5), f(x2)=(0.2, −0.1)f(x_2)=(0.2,\,-0.1) and m=1.5m=1.5. Then f(x1)−f(x2)=(0.8, 0.6)f(x_1)-f(x_2)=(0.8,\,0.6) and D=1.0D=1.0. A second pair with f(x2)=(−0.6, −0.7)f(x_2)=(-0.6,\,-0.7) has difference (1.6, 1.2)(1.6,\,1.2) and D=2.0D=2.0.

pairlabelL∂L/∂D∂L/∂f(x₁)
D=1.0D=1.0similar12(1.0)2=0.5\tfrac12(1.0)^2=0.51.01.0(0.8, 0.6)(0.8,\,0.6)
D=1.0D=1.0dissimilar12(1.5−1.0)2=0.125\tfrac12(1.5-1.0)^2=0.125−0.5-0.5(−0.4, −0.3)(-0.4,\,-0.3)
D=2.0D=2.0similar12(2.0)2=2.0\tfrac12(2.0)^2=2.02.02.0(1.6, 1.2)(1.6,\,1.2)
D=2.0D=2.0dissimilarmax⁡(0,−0.5)2/2=0\max(0,-0.5)^2/2=000(0, 0)(0,\,0)
Loss and gradient for each case (m = 1.5), checked with the figure's code in Node

Reading the dissimilar D = 1.0 row

  1. 1The pair is closer than the margin by m−D=0.5m-D=0.5, so the hinge is active and L=12⋅0.52=0.125L=\tfrac12\cdot0.5^2=0.125.
  2. 2∂L/∂D=−(m−D)=−0.5\partial L/\partial D=-(m-D)=-0.5: increasing the distance lowers the loss.
  3. 3Multiply by the unit vector (0.8,0.6)/1.0(0.8,0.6)/1.0: ∂L/∂f(x1)=(−0.4,−0.3)\partial L/\partial f(x_1)=(-0.4,-0.3) and ∂L/∂f(x2)=(0.4,0.3)\partial L/\partial f(x_2)=(0.4,0.3).
  4. 4Gradient descent moves f(x1)f(x_1) along +(0.4,0.3)+(0.4,0.3) and f(x2)f(x_2) along −(0.4,0.3)-(0.4,0.3): straight apart. Backprop then carries these vectors into the shared weights from both branches.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3def contrastive_loss(z1, z2, y, margin=1.0):
4 """z1, z2: (B, d) embeddings; y: (B,) float, 1 = similar, 0 = dissimilar."""
5 d = F.pairwise_distance(z1, z2) # ||z1 - z2 + eps||_2, eps = 1e-6
6 pos = y * d.pow(2)
7 neg = (1 - y) * F.relu(margin - d).pow(2)
8 return 0.5 * (pos + neg).mean()
9
10z1 = torch.tensor([[1.0, 0.5], [1.0, 0.5]], requires_grad=True)
11z2 = torch.tensor([[0.2, -0.1], [0.2, -0.1]])
12y = torch.tensor([1.0, 0.0])
13loss = contrastive_loss(z1, z2, y, margin=1.5) # (0.5 + 0.125) / 2 = 0.3125 (up to eps)
14
15# Siamese training: ONE encoder applied to both inputs, so gradients from both branches add up
16encoder = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 32))
17x1, x2 = torch.randn(64, 784), torch.randn(64, 784)
18same = torch.randint(0, 2, (64,)).float()
19loss = contrastive_loss(encoder(x1), encoder(x2), same, margin=1.0)
20loss.backward()
21
22# Built-in, unsquared: input is the distance, target is +1 (similar) / -1 (dissimilar)
23hinge = nn.HingeEmbeddingLoss(margin=1.0) # x if y = 1, max(0, margin - x) if y = -1
24loss2 = hinge(F.pairwise_distance(encoder(x1), encoder(x2)), 2 * same - 1)
Contrastive loss in PyTorch (siamese setup), plus the built-in unsquared cousin

Use when

  • +Needs only same / different labels, no fixed class list: works for verification and open-set retrieval
  • +Simple, smooth, two-line implementation with spring-like, interpretable forces
  • +The margin bounds the push, so the embedding cannot win by inflating its scale
  • +Each pair is independent: easy to batch and to combine with other losses

Avoid when

  • −Absolute targets (0 and ≥ m) for every pair, even for classes that are naturally spread out
  • −Most random negatives give zero gradient; needs balanced sampling or hard-negative mining
  • −Margin must match the embedding scale; collapse of a class to one point is encouraged
  • −Looks at one negative at a time, so it learns more slowly than multi-negative losses

Check yourself

With margin m = 1.5, a dissimilar pair has D = 2.0. What does it contribute to the gradient?

Check yourself

Why can't you train an embedding with only the similar-pair term ½D²?

Remember

  • ✓L=y 12D2+(1−y) 12max⁡(0,m−D)2L=y\,\tfrac12D^2+(1-y)\,\tfrac12\max(0,m-D)^2 on pairs, with DD the distance between shared-network embeddings (Hadsell, Chopra and LeCun, 2006).
  • ✓Gradient on each embedding is ∂L/∂D\partial L/\partial D times the unit vector along the pair: a spring pull DD, or a push m−Dm-D that stops at the margin.
  • ✓Dissimilar pairs beyond the margin give exactly zero gradient, so sampling and hard-negative mining matter.
  • ✓The margin is meaningful only relative to the embedding scale; check the label convention (DrLIM uses Y = 0 for similar).
  • ✓It learns a metric, which is what verification and retrieval with unseen classes need.
← back to Deep Learning roadmap