Definition
Contrastive loss (pairwise margin loss) is a loss on a pair of inputs and a binary label saying whether they are similar. With embeddings and their distance , a similar pair pays and a dissimilar pair pays . Minimising it makes distance in embedding space mean "how different are these".
Ordinary classification losses need a fixed set of classes. Many problems do not have one: verifying that two signatures or two faces belong to the same person, finding near-duplicate images, or learning a low-dimensional map where neighbours stay neighbours. What you do have is pairs labelled same or different. Contrastive loss turns that weak supervision into a metric: the network is trained so that the Euclidean distance between its outputs reflects the labels, and new classes need no retraining.
The idea grew out of siamese networks: two copies of one network with shared weights, compared at the output. Bromley et al. (1993) used one for signature verification, and Chopra, Hadsell and LeCun (2005) trained a convolutional siamese net for face verification with an energy-based pair loss. The form used today comes from Hadsell, Chopra and LeCun, Dimensionality Reduction by Learning an Invariant Mapping (DrLIM, CVPR 2006). They described it as a system of springs: similar pairs are joined by attract-only springs, dissimilar pairs by repulse-only springs that go slack once the pair is at least apart.
Formula and gradient
- 1 for a similar (positive) pair, 0 for a dissimilar (negative) pair. DrLIM writes the opposite convention, for similar
- Euclidean distance between the two embeddings (not squared)
- the margin: dissimilar pairs with contribute zero loss and zero gradient
- the shared embedding network (one set of weights, applied to both inputs)
- unit vector along the pair: the force always acts along the line joining the two points
- pull strength on a similar pair: grows linearly with distance, like a spring
- push strength on a dissimilar pair: largest when the two coincide, zero at the margin
The two halves are equal and opposite on the two embeddings, so each pair moves symmetrically. A similar pair gradient is exactly the MSE gradient between two learned vectors. The dissimilar half is a squared hinge on distance: it switches on only when the pair is closer than . Without that switch, the loss could be lowered forever by pushing negatives to infinity, and without negatives at all the trivial solution would give zero loss.
At a glance
- Input
- pairs with a binary label similar / dissimilar
- Output
- an embedding network ; similarity is read off as distance
- Range
- for similar pairs, for dissimilar pairs
- Zero loss when
- similar pairs coincide and dissimilar pairs are at least apart
- Hyperparameter
- margin ; it only has meaning relative to the scale of (fix the scale by L2-normalising, or let the loss set it)
- PyTorch
- no
nn.ContrastiveLoss; write it in two lines, or usenn.HingeEmbeddingLossfor the unsquared variant - Successors
- triplet loss (relative distances) and InfoNCE (many negatives at once)
Why the margin, and why it is squared
Think of the dissimilar term as a spring that can only push. Its force is at distance and nothing beyond, so a negative pair is only ever moved out to the margin, never further. That keeps the loss bounded below, stops the embedding from inflating its scale to make every negative trivially far, and focuses learning on the confusable pairs. In the figure, most of the 9 negative pairs begin outside the margin and get no gradient at all.
Squaring both terms makes the forces continuous: the pull fades to 0 as a similar pair meets, and the push fades to 0 as a negative reaches the margin, so there is no kink to bounce around. Unsquared variants exist. PyTorch's nn.HingeEmbeddingLoss uses for similar pairs and for dissimilar ones, which gives constant-magnitude forces instead of spring-like ones.
Worked example: one pair, both labels
Take , and . Then and . A second pair with has difference and .
| pair | label | L | ∂L/∂D | ∂L/∂f(x₁) |
|---|---|---|---|---|
| similar | ||||
| dissimilar | ||||
| similar | ||||
| dissimilar |
Reading the dissimilar D = 1.0 row
- 1The pair is closer than the margin by , so the hinge is active and .
- 2: increasing the distance lowers the loss.
- 3Multiply by the unit vector : and .
- 4Gradient descent moves along and along : straight apart. Backprop then carries these vectors into the shared weights from both branches.
1import torch, torch.nn as nn, torch.nn.functional as F23def contrastive_loss(z1, z2, y, margin=1.0):4 """z1, z2: (B, d) embeddings; y: (B,) float, 1 = similar, 0 = dissimilar."""5 d = F.pairwise_distance(z1, z2) # ||z1 - z2 + eps||_2, eps = 1e-66 pos = y * d.pow(2)7 neg = (1 - y) * F.relu(margin - d).pow(2)8 return 0.5 * (pos + neg).mean()910z1 = torch.tensor([[1.0, 0.5], [1.0, 0.5]], requires_grad=True)11z2 = torch.tensor([[0.2, -0.1], [0.2, -0.1]])12y = torch.tensor([1.0, 0.0])13loss = contrastive_loss(z1, z2, y, margin=1.5) # (0.5 + 0.125) / 2 = 0.3125 (up to eps)1415# Siamese training: ONE encoder applied to both inputs, so gradients from both branches add up16encoder = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 32))17x1, x2 = torch.randn(64, 784), torch.randn(64, 784)18same = torch.randint(0, 2, (64,)).float()19loss = contrastive_loss(encoder(x1), encoder(x2), same, margin=1.0)20loss.backward()2122# Built-in, unsquared: input is the distance, target is +1 (similar) / -1 (dissimilar)23hinge = nn.HingeEmbeddingLoss(margin=1.0) # x if y = 1, max(0, margin - x) if y = -124loss2 = hinge(F.pairwise_distance(encoder(x1), encoder(x2)), 2 * same - 1)Use when
- +Needs only same / different labels, no fixed class list: works for verification and open-set retrieval
- +Simple, smooth, two-line implementation with spring-like, interpretable forces
- +The margin bounds the push, so the embedding cannot win by inflating its scale
- +Each pair is independent: easy to batch and to combine with other losses
Avoid when
- −Absolute targets (0 and ≥ m) for every pair, even for classes that are naturally spread out
- −Most random negatives give zero gradient; needs balanced sampling or hard-negative mining
- −Margin must match the embedding scale; collapse of a class to one point is encouraged
- −Looks at one negative at a time, so it learns more slowly than multi-negative losses
Check yourself
With margin m = 1.5, a dissimilar pair has D = 2.0. What does it contribute to the gradient?
Check yourself
Why can't you train an embedding with only the similar-pair term ½D²?
Remember
- ✓ on pairs, with the distance between shared-network embeddings (Hadsell, Chopra and LeCun, 2006).
- ✓Gradient on each embedding is times the unit vector along the pair: a spring pull , or a push that stops at the margin.
- ✓Dissimilar pairs beyond the margin give exactly zero gradient, so sampling and hard-negative mining matter.
- ✓The margin is meaningful only relative to the embedding scale; check the label convention (DrLIM uses Y = 0 for similar).
- ✓It learns a metric, which is what verification and retrieval with unseen classes need.