Cosine embedding loss

A pair loss on angles, not distances: 1 − cos for similar pairs, max(0, cos − m) for dissimilar ones. Vector length is ignored, the gradient is perpendicular to each vector and shrinks as 1/‖x‖, which is why it is usually paired with normalisation.

10 min read683 wordsadvanced0 viewsupdated 2026-10-11

Definition

Cosine embedding loss is a pairwise loss on the cosine similarity cos⁡(x1,x2)=x1⊤x2∥x1∥∥x2∥\cos(x_1,x_2)=\frac{x_1^\top x_2}{\lVert x_1\rVert\lVert x_2\rVert} of two embeddings with a label y∈{+1,−1}y\in\{+1,-1\}: a similar pair pays 1−cos⁡1-\cos, a dissimilar pair pays max⁡(0,cos⁡−m)\max(0,\cos-m). It is the angular counterpart of contrastive loss, and PyTorch ships it as nn.CosineEmbeddingLoss.

In many embedding spaces the direction of a vector carries the meaning and its length carries something else: word frequency, image contrast, model confidence. Retrieval with dot products on normalised vectors, sentence similarity and cross-modal matching all compare angles. If the comparison at test time is the cosine, it makes sense to train on the cosine too. The cosine embedding loss does exactly that, with the same similar/dissimilar pair supervision and the same margin idea as the contrastive loss of Hadsell, Chopra and LeCun (2006), but measured on the sphere instead of in Euclidean space.

It is a convenience loss rather than a named paper: Torch7's nn.CosineEmbeddingCriterion already had this form, and PyTorch kept it. Its value is less in novelty than in its geometry, which explains several behaviours you meet with any cosine-based objective, including InfoNCE with cosine scores and the logits of cosine classifiers.

Formula and gradient

L(x1,x2,y)={1−cos⁡(x1,x2)y=+1max⁡(0, cos⁡(x1,x2)−m)y=−1cos⁡(x1,x2)=x1⊤x2∥x1∥ ∥x2∥L(x_1,x_2,y)=\begin{cases}1-\cos(x_1,x_2) & y=+1\\ \max\big(0,\ \cos(x_1,x_2)-m\big) & y=-1\end{cases}\qquad \cos(x_1,x_2)=\frac{x_1^\top x_2}{\lVert x_1\rVert\,\lVert x_2\rVert}
yy
+1+1 similar, −1-1 dissimilar (PyTorch's convention; not 0/1)
mm
margin in [−1,1][-1,1]; PyTorch's default is 0 and its docs suggest 0 to 0.5. A dissimilar pair with cos⁡≤m\cos\le m (angle at least arccos⁡m\arccos m) costs nothing
LL
in [0,2][0,2] for similar pairs and [0,1−m][0,1-m] for dissimilar ones
∂cos⁡∂x1=1∥x1∥(x^2−cos⁡θ x^1),∥∂cos⁡∂x1∥=sin⁡θ∥x1∥,x1⊤∂cos⁡∂x1=0\frac{\partial\cos}{\partial x_1}=\frac{1}{\lVert x_1\rVert}\Big(\hat x_2-\cos\theta\,\hat x_1\Big),\qquad \Big\lVert\frac{\partial\cos}{\partial x_1}\Big\rVert=\frac{\sin\theta}{\lVert x_1\rVert},\qquad x_1^\top\frac{\partial\cos}{\partial x_1}=0
x^=x/∥x∥\hat x=x/\lVert x\rVert
unit vector in the direction of x
x^2−cos⁡θ x^1\hat x_2-\cos\theta\,\hat x_1
the part of x^2\hat x_2 perpendicular to x1x_1: the gradient only rotates x1x_1
1/∥x1∥1/\lVert x_1\rVert
longer vectors get proportionally smaller gradients for the same angle
∂L/∂x1\partial L/\partial x_1
−∂cos⁡/∂x1-\partial\cos/\partial x_1 for y=+1y=+1; +∂cos⁡/∂x1+\partial\cos/\partial x_1 for an active dissimilar pair; 0 once cos⁡≤m\cos\le m

Two consequences follow directly. First, because cosine is invariant to scaling, its gradient has no radial component: the loss cannot change vector lengths except as a side effect. Second, the gradient magnitude is sin⁡θ/∥x1∥\sin\theta/\lVert x_1\rVert, so it vanishes for parallel (θ=0\theta=0) and anti-parallel (θ=180°\theta=180°) pairs, and it is small for long vectors. A similar pair pointing in opposite directions has the largest loss, 2, and almost no gradient.

At a glance

Input
two batches x1,x2∈RN×Dx_1,x_2\in\mathbb R^{N\times D} and y∈{+1,−1}Ny\in\{+1,-1\}^N
Measures
the angle θ\theta between embeddings; invariant to x↦cxx\mapsto cx for c>0c>0
Range
[0,2][0,2] for similar pairs, [0,1−m][0,1-m] for dissimilar pairs
Gradient
perpendicular to the vector, length sin⁡θ/∥x∥\sin\theta/\lVert x\rVert
Margin
only on the dissimilar side; m=0m=0 asks dissimilar pairs to be at least orthogonal
PyTorch
nn.CosineEmbeddingLoss(margin=0.0, reduction="mean"); adds ε=10−12\varepsilon=10^{-12} to each squared norm
drag x₁, x₂ · rose arrows = −∂L/∂xx₁x₂dashed circle: ‖x‖ = 1L vs angle θ between x₁ and x₂0°45°90°135°180°012arccos m = 78.5°y=+1: 1 − cos θy=−1: max(0, cos θ − m)cos θ = 0.537 · θ = 57.5°L = 0.463 ‖x₁‖ = 1.649 · ‖x₂‖ = 1.897‖∂L/∂x₁‖ = 0.5115 = sin θ / ‖x₁‖‖∂L/∂x₂‖ = 0.4446 · radial part 0scaling x₁ leaves L unchanged but scales its gradient by 1/‖x₁‖: long vectors learn slowly
label y
fig.Drag the two vectors. At the start x₁ = (1.6, 0.4) and x₂ = (0.6, 1.8): cos θ = 0.537, θ = 57.5°, and for y = +1 the loss is 0.463. The rose arrows (−∂L/∂x) are tangent to circles around the origin, and the readout shows a radial part of 0. ‖∂L/∂x₁‖ = 0.5115 = sin θ / ‖x₁‖. Press ‖x₁‖ ×2: the loss stays 0.463 but ‖∂L/∂x₁‖ halves to 0.2558. Switch to y = −1 with m = 0.2: the loss is cos θ − m = 0.337, the gradient flips to push the vectors apart, and the shaded cone (cos > m, half-angle arccos 0.2 = 78.5°) shows where a dissimilar x₂ is still penalised.

The geometry: rotation only, and a shrinking step

A gradient step perpendicular to xx always makes xx longer: ∥x−ηg∥2=∥x∥2+η2∥g∥2\lVert x-\eta g\rVert^2=\lVert x\rVert^2+\eta^2\lVert g\rVert^2 when g⊥xg\perp x. So with plain SGD on free embedding vectors, the norms creep up step after step, and since the gradient scales as 1/∥x∥1/\lVert x\rVert, the effective learning rate on the angle falls over training. In the worked example below one step of size 1 lengthens x1x_1 from 2 to 2.031. Weight decay counteracts the growth; normalising the embedding (x←x/∥x∥x\leftarrow x/\lVert x\rVert) removes it.

ba
a = (3, 1)b = (1, 2)a·b = 5.00θ = 45.0°
fig.The dot product behind the cosine. Drag either arrow: a·b = ‖a‖‖b‖cos θ, so the cosine is the dot product with both lengths divided out. At the start a = (3, 1), b = (1, 2), a·b = 5 and θ = 45°.

Worked example

Let x1=(2,0)x_1=(2,0), x2=(1,1)x_2=(1,1), so cos⁡θ=2/(22)=0.7071\cos\theta=2/(2\sqrt2)=0.7071 and θ=45°\theta=45°. Use m=0.5m=0.5 for the dissimilar case.

Similar pair (y = +1)

  1. 1L=1−0.7071=0.2929L=1-0.7071=0.2929.
  2. 2∂cos⁡/∂x1=x2∥x1∥∥x2∥−cos⁡θ x1∥x1∥2=(0.3536,0.3536)−(0.3536,0)=(0, 0.3536)\partial\cos/\partial x_1=\frac{x_2}{\lVert x_1\rVert\lVert x_2\rVert}-\cos\theta\,\frac{x_1}{\lVert x_1\rVert^2}=(0.3536,0.3536)-(0.3536,0)=(0,\,0.3536): perpendicular to x1=(2,0)x_1=(2,0), length sin⁡45°/2=0.3536\sin45°/2=0.3536.
  3. 3∂L/∂x1=−(0,0.3536)=(0,−0.3536)\partial L/\partial x_1=-(0,0.3536)=(0,-0.3536), so descent moves x1x_1 toward +y+y, i.e. rotates it toward x2x_2.
  4. 4For x2x_2: ∂L/∂x2=(−0.3536, 0.3536)\partial L/\partial x_2=(-0.3536,\,0.3536), perpendicular to (1,1)(1,1); descent rotates x2x_2 toward x1x_1.
  5. 5One step with η=1\eta=1 takes x1x_1 to (2, 0.3536)(2,\,0.3536), whose length is 2.0312.031: the angle shrank and the norm grew.
x₁yL∂L/∂x₁‖∂L/∂x₁‖
(2, 0)+10.2929(0, −0.3536)0.3536
(2, 0)−10.7071−0.5=0.20710.7071-0.5=0.2071(0, +0.3536)0.3536
(4, 0)+10.2929(0, −0.1768)0.1768
(4, 0)−10.2071(0, +0.1768)0.1768
Same angle, different lengths and labels (m = 0.5), checked in Node
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x1 = torch.tensor([[2.0, 0.0], [2.0, 0.0]], requires_grad=True)
4x2 = torch.tensor([[1.0, 1.0], [1.0, 1.0]])
5y = torch.tensor([1.0, -1.0]) # +1 similar, -1 dissimilar
6
7loss_fn = nn.CosineEmbeddingLoss(margin=0.5) # reduction="mean" over the batch
8loss = loss_fn(x1, x2, y) # (0.2929 + 0.2071) / 2 = 0.25
9loss.backward()
10print(x1.grad) # [[0, -0.1768], [0, 0.1768]] (each row / batch size 2)
11
12# Same thing by hand
13cos = F.cosine_similarity(x1, x2, dim=1)
14manual = torch.where(y > 0, 1 - cos, F.relu(cos - 0.5)).mean()
15
16# Typical use: two encoders whose outputs should agree in direction
17img_enc, txt_enc = nn.Linear(512, 128), nn.Linear(300, 128)
18img, txt = torch.randn(64, 512), torch.randn(64, 300)
19match = torch.where(torch.rand(64) < 0.5, 1.0, -1.0)
20loss = nn.CosineEmbeddingLoss(margin=0.2)(img_enc(img), txt_enc(txt), match)
nn.CosineEmbeddingLoss and its two-line equivalent

Use when

  • +Matches cosine-similarity retrieval, so training and test measure the same thing
  • +Scale invariant: insensitive to embedding norms from frequency, contrast or confidence
  • +Bounded loss (at most 2), robust to outlier vectors with huge norms
  • +Built into PyTorch; same pair supervision as contrastive loss

Avoid when

  • −Gradient shrinks as 1/∥x∥1/\lVert x\rVert and vanishes at θ=0°\theta=0° and 180°180°
  • −SGD steps lengthen unnormalised embeddings, silently lowering the effective learning rate
  • −Like any pair loss: one negative per term, most random negatives give zero gradient
  • −±1 labels and the margin's meaning (a cosine, not a distance) are easy to get wrong

Check yourself

You double the length of x₁ without changing its direction. What happens to the cosine embedding loss and its gradient with respect to x₁?

Check yourself

Which direction does the gradient ∂L/∂x₁ of a cosine loss point?

Remember

  • ✓L=1−cos⁡L=1-\cos for y=+1y=+1, max⁡(0,cos⁡−m)\max(0,\cos-m) for y=−1y=-1: contrastive loss measured in angles (nn.CosineEmbeddingLoss).
  • ✓Gradient 1∥x∥(x^2−cos⁡θ x^1)\frac{1}{\lVert x\rVert}(\hat x_2-\cos\theta\,\hat x_1): perpendicular to xx, size sin⁡θ/∥x∥\sin\theta/\lVert x\rVert.
  • ✓Lengths are ignored by the loss but grow under SGD, which slowly shrinks the effective step: normalise or use weight decay.
  • ✓On unit vectors, 1−cos⁡θ=12∥x^1−x^2∥21-\cos\theta=\tfrac12\lVert\hat x_1-\hat x_2\rVert^2, so it is the squared-distance loss on the sphere.
  • ✓Labels must be ±1; the margin is a cosine in [−1, 1], with 0 to 0.5 typical.
← back to Deep Learning roadmap