Definition
Cosine embedding loss is a pairwise loss on the cosine similarity of two embeddings with a label : a similar pair pays , a dissimilar pair pays . It is the angular counterpart of contrastive loss, and PyTorch ships it as nn.CosineEmbeddingLoss.
In many embedding spaces the direction of a vector carries the meaning and its length carries something else: word frequency, image contrast, model confidence. Retrieval with dot products on normalised vectors, sentence similarity and cross-modal matching all compare angles. If the comparison at test time is the cosine, it makes sense to train on the cosine too. The cosine embedding loss does exactly that, with the same similar/dissimilar pair supervision and the same margin idea as the contrastive loss of Hadsell, Chopra and LeCun (2006), but measured on the sphere instead of in Euclidean space.
It is a convenience loss rather than a named paper: Torch7's nn.CosineEmbeddingCriterion already had this form, and PyTorch kept it. Its value is less in novelty than in its geometry, which explains several behaviours you meet with any cosine-based objective, including InfoNCE with cosine scores and the logits of cosine classifiers.
Formula and gradient
- similar, dissimilar (PyTorch's convention; not 0/1)
- margin in ; PyTorch's default is 0 and its docs suggest 0 to 0.5. A dissimilar pair with (angle at least ) costs nothing
- in for similar pairs and for dissimilar ones
- unit vector in the direction of x
- the part of perpendicular to : the gradient only rotates
- longer vectors get proportionally smaller gradients for the same angle
- for ; for an active dissimilar pair; 0 once
Two consequences follow directly. First, because cosine is invariant to scaling, its gradient has no radial component: the loss cannot change vector lengths except as a side effect. Second, the gradient magnitude is , so it vanishes for parallel () and anti-parallel () pairs, and it is small for long vectors. A similar pair pointing in opposite directions has the largest loss, 2, and almost no gradient.
At a glance
- Input
- two batches and
- Measures
- the angle between embeddings; invariant to for
- Range
- for similar pairs, for dissimilar pairs
- Gradient
- perpendicular to the vector, length
- Margin
- only on the dissimilar side; asks dissimilar pairs to be at least orthogonal
- PyTorch
nn.CosineEmbeddingLoss(margin=0.0, reduction="mean"); adds to each squared norm
The geometry: rotation only, and a shrinking step
A gradient step perpendicular to always makes longer: when . So with plain SGD on free embedding vectors, the norms creep up step after step, and since the gradient scales as , the effective learning rate on the angle falls over training. In the worked example below one step of size 1 lengthens from 2 to 2.031. Weight decay counteracts the growth; normalising the embedding () removes it.
Worked example
Let , , so and . Use for the dissimilar case.
Similar pair (y = +1)
- 1.
- 2: perpendicular to , length .
- 3, so descent moves toward , i.e. rotates it toward .
- 4For : , perpendicular to ; descent rotates toward .
- 5One step with takes to , whose length is : the angle shrank and the norm grew.
| x₁ | y | L | ∂L/∂x₁ | ‖∂L/∂x₁‖ |
|---|---|---|---|---|
| (2, 0) | +1 | 0.2929 | (0, −0.3536) | 0.3536 |
| (2, 0) | −1 | (0, +0.3536) | 0.3536 | |
| (4, 0) | +1 | 0.2929 | (0, −0.1768) | 0.1768 |
| (4, 0) | −1 | 0.2071 | (0, +0.1768) | 0.1768 |
1import torch, torch.nn as nn, torch.nn.functional as F23x1 = torch.tensor([[2.0, 0.0], [2.0, 0.0]], requires_grad=True)4x2 = torch.tensor([[1.0, 1.0], [1.0, 1.0]])5y = torch.tensor([1.0, -1.0]) # +1 similar, -1 dissimilar67loss_fn = nn.CosineEmbeddingLoss(margin=0.5) # reduction="mean" over the batch8loss = loss_fn(x1, x2, y) # (0.2929 + 0.2071) / 2 = 0.259loss.backward()10print(x1.grad) # [[0, -0.1768], [0, 0.1768]] (each row / batch size 2)1112# Same thing by hand13cos = F.cosine_similarity(x1, x2, dim=1)14manual = torch.where(y > 0, 1 - cos, F.relu(cos - 0.5)).mean()1516# Typical use: two encoders whose outputs should agree in direction17img_enc, txt_enc = nn.Linear(512, 128), nn.Linear(300, 128)18img, txt = torch.randn(64, 512), torch.randn(64, 300)19match = torch.where(torch.rand(64) < 0.5, 1.0, -1.0)20loss = nn.CosineEmbeddingLoss(margin=0.2)(img_enc(img), txt_enc(txt), match)Use when
- +Matches cosine-similarity retrieval, so training and test measure the same thing
- +Scale invariant: insensitive to embedding norms from frequency, contrast or confidence
- +Bounded loss (at most 2), robust to outlier vectors with huge norms
- +Built into PyTorch; same pair supervision as contrastive loss
Avoid when
- −Gradient shrinks as and vanishes at and
- −SGD steps lengthen unnormalised embeddings, silently lowering the effective learning rate
- −Like any pair loss: one negative per term, most random negatives give zero gradient
- −±1 labels and the margin's meaning (a cosine, not a distance) are easy to get wrong
Check yourself
You double the length of x₁ without changing its direction. What happens to the cosine embedding loss and its gradient with respect to x₁?
Check yourself
Which direction does the gradient ∂L/∂x₁ of a cosine loss point?
Remember
- ✓ for , for : contrastive loss measured in angles (
nn.CosineEmbeddingLoss). - ✓Gradient : perpendicular to , size .
- ✓Lengths are ignored by the loss but grow under SGD, which slowly shrinks the effective step: normalise or use weight decay.
- ✓On unit vectors, , so it is the squared-distance loss on the sphere.
- ✓Labels must be ±1; the margin is a cosine in [−1, 1], with 0 to 0.5 typical.