# Cosine embedding loss

> A pair loss on angles, not distances: 1 − cos for similar pairs, max(0, cos − m) for dissimilar ones. Vector length is ignored, the gradient is perpendicular to each vector and shrinks as 1/‖x‖, which is why it is usually paired with normalisation.

Source: https://www.mlfoundry.in/learn/cosine-embedding-loss  
Section: Deep Learning › Loss Functions  
Level: advanced · 10 min read · updated 2026-10-11

**Cosine embedding loss:** is a pairwise loss on the **cosine similarity** $\cos(x_1,x_2)=\frac{x_1^\top x_2}{\lVert x_1\rVert\lVert x_2\rVert}$ of two embeddings with a label $y\in\{+1,-1\}$: a similar pair pays $1-\cos$, a dissimilar pair pays $\max(0,\cos-m)$. It is the angular counterpart of contrastive loss, and PyTorch ships it as `nn.CosineEmbeddingLoss`.

In many embedding spaces the **direction** of a vector carries the meaning and its length carries something else: word frequency, image contrast, model confidence. Retrieval with dot products on normalised vectors, sentence similarity and cross-modal matching all compare angles. If the comparison at test time is the cosine, it makes sense to train on the cosine too. The cosine embedding loss does exactly that, with the same similar/dissimilar pair supervision and the same margin idea as the contrastive loss of Hadsell, Chopra and LeCun (2006), but measured on the sphere instead of in Euclidean space.

It is a convenience loss rather than a named paper: Torch7's `nn.CosineEmbeddingCriterion` already had this form, and PyTorch kept it. Its value is less in novelty than in its geometry, which explains several behaviours you meet with any cosine-based objective, including InfoNCE with cosine scores and the logits of cosine classifiers.

## Formula and gradient

$$
L(x_1,x_2,y)=\begin{cases}1-\cos(x_1,x_2) & y=+1\\ \max\big(0,\ \cos(x_1,x_2)-m\big) & y=-1\end{cases}\qquad \cos(x_1,x_2)=\frac{x_1^\top x_2}{\lVert x_1\rVert\,\lVert x_2\rVert}
$$

- $y$: $+1$ similar, $-1$ dissimilar (PyTorch's convention; not 0/1)
- $m$: margin in $[-1,1]$; PyTorch's default is 0 and its docs suggest 0 to 0.5. A dissimilar pair with $\cos\le m$ (angle at least $\arccos m$) costs nothing
- $L$: in $[0,2]$ for similar pairs and $[0,1-m]$ for dissimilar ones

$$
\frac{\partial\cos}{\partial x_1}=\frac{1}{\lVert x_1\rVert}\Big(\hat x_2-\cos\theta\,\hat x_1\Big),\qquad \Big\lVert\frac{\partial\cos}{\partial x_1}\Big\rVert=\frac{\sin\theta}{\lVert x_1\rVert},\qquad x_1^\top\frac{\partial\cos}{\partial x_1}=0
$$

- $\hat x=x/\lVert x\rVert$: unit vector in the direction of x
- $\hat x_2-\cos\theta\,\hat x_1$: the part of $\hat x_2$ perpendicular to $x_1$: the gradient only rotates $x_1$
- $1/\lVert x_1\rVert$: longer vectors get proportionally smaller gradients for the same angle
- $\partial L/\partial x_1$: $-\partial\cos/\partial x_1$ for $y=+1$; $+\partial\cos/\partial x_1$ for an active dissimilar pair; 0 once $\cos\le m$

Two consequences follow directly. First, because cosine is invariant to scaling, its gradient has **no radial component**: the loss cannot change vector lengths except as a side effect. Second, the gradient magnitude is $\sin\theta/\lVert x_1\rVert$, so it vanishes for parallel ($\theta=0$) and anti-parallel ($\theta=180°$) pairs, and it is small for long vectors. A similar pair pointing in opposite directions has the largest loss, 2, and almost no gradient.

**At a glance**

- **Input:** two batches $x_1,x_2\in\mathbb R^{N\times D}$ and $y\in\{+1,-1\}^N$
- **Measures:** the angle $\theta$ between embeddings; invariant to $x\mapsto cx$ for $c>0$
- **Range:** $[0,2]$ for similar pairs, $[0,1-m]$ for dissimilar pairs
- **Gradient:** perpendicular to the vector, length $\sin\theta/\lVert x\rVert$
- **Margin:** only on the dissimilar side; $m=0$ asks dissimilar pairs to be at least orthogonal
- **PyTorch:** `nn.CosineEmbeddingLoss(margin=0.0, reduction="mean")`; adds $\varepsilon=10^{-12}$ to each squared norm

> Figure: Drag the two vectors. At the start x₁ = (1.6, 0.4) and x₂ = (0.6, 1.8): cos θ = 0.537, θ = 57.5°, and for y = +1 the loss is 0.463. The rose arrows (−∂L/∂x) are tangent to circles around the origin, and the readout shows a radial part of 0. ‖∂L/∂x₁‖ = 0.5115 = sin θ / ‖x₁‖. Press ‖x₁‖ ×2: the loss stays 0.463 but ‖∂L/∂x₁‖ halves to 0.2558. Switch to y = −1 with m = 0.2: the loss is cos θ − m = 0.337, the gradient flips to push the vectors apart, and the shaded cone (cos > m, half-angle arccos 0.2 = 78.5°) shows where a dissimilar x₂ is still penalised.

## The geometry: rotation only, and a shrinking step

A gradient step perpendicular to $x$ always makes $x$ longer: $\lVert x-\eta g\rVert^2=\lVert x\rVert^2+\eta^2\lVert g\rVert^2$ when $g\perp x$. So with plain SGD on free embedding vectors, the norms creep up step after step, and since the gradient scales as $1/\lVert x\rVert$, the **effective learning rate on the angle falls** over training. In the worked example below one step of size 1 lengthens $x_1$ from 2 to 2.031. Weight decay counteracts the growth; normalising the embedding ($x\leftarrow x/\lVert x\rVert$) removes it.

> **Normalise, then the loss is a distance loss:** For unit vectors $\lVert\hat x_1-\hat x_2\rVert^2=2-2\cos\theta$, so $1-\cos\theta=\tfrac12\lVert\hat x_1-\hat x_2\rVert^2$. On the sphere, the similar-pair cosine loss **is** half the squared Euclidean distance, and the dissimilar hinge $\cos\theta\le m$ is the distance condition $\lVert\hat x_1-\hat x_2\rVert\ge\sqrt{2-2m}$. Cosine and Euclidean losses differ mainly in what they do to the vector lengths.

> Figure: The dot product behind the cosine. Drag either arrow: a·b = ‖a‖‖b‖cos θ, so the cosine is the dot product with both lengths divided out. At the start a = (3, 1), b = (1, 2), a·b = 5 and θ = 45°.

## Worked example

Let $x_1=(2,0)$, $x_2=(1,1)$, so $\cos\theta=2/(2\sqrt2)=0.7071$ and $\theta=45°$. Use $m=0.5$ for the dissimilar case.

**Similar pair (y = +1)**

1. $L=1-0.7071=0.2929$.
2. $\partial\cos/\partial x_1=\frac{x_2}{\lVert x_1\rVert\lVert x_2\rVert}-\cos\theta\,\frac{x_1}{\lVert x_1\rVert^2}=(0.3536,0.3536)-(0.3536,0)=(0,\,0.3536)$: perpendicular to $x_1=(2,0)$, length $\sin45°/2=0.3536$.
3. $\partial L/\partial x_1=-(0,0.3536)=(0,-0.3536)$, so descent moves $x_1$ toward $+y$, i.e. rotates it toward $x_2$.
4. For $x_2$: $\partial L/\partial x_2=(-0.3536,\,0.3536)$, perpendicular to $(1,1)$; descent rotates $x_2$ toward $x_1$.
5. One step with $\eta=1$ takes $x_1$ to $(2,\,0.3536)$, whose length is $2.031$: the angle shrank and the norm grew.

Same angle, different lengths and labels (m = 0.5), checked in Node

| x₁ | y | L | ∂L/∂x₁ | ‖∂L/∂x₁‖ |
| --- | --- | --- | --- | --- |
| (2, 0) | +1 | 0.2929 | (0, −0.3536) | 0.3536 |
| (2, 0) | −1 | $0.7071-0.5=0.2071$ | (0, +0.3536) | 0.3536 |
| (4, 0) | +1 | 0.2929 | (0, −0.1768) | 0.1768 |
| (4, 0) | −1 | 0.2071 | (0, +0.1768) | 0.1768 |

```python
import torch, torch.nn as nn, torch.nn.functional as F

x1 = torch.tensor([[2.0, 0.0], [2.0, 0.0]], requires_grad=True)
x2 = torch.tensor([[1.0, 1.0], [1.0, 1.0]])
y = torch.tensor([1.0, -1.0])                       # +1 similar, -1 dissimilar

loss_fn = nn.CosineEmbeddingLoss(margin=0.5)       # reduction="mean" over the batch
loss = loss_fn(x1, x2, y)                           # (0.2929 + 0.2071) / 2 = 0.25
loss.backward()
print(x1.grad)                                      # [[0, -0.1768], [0, 0.1768]]  (each row / batch size 2)

# Same thing by hand
cos = F.cosine_similarity(x1, x2, dim=1)
manual = torch.where(y > 0, 1 - cos, F.relu(cos - 0.5)).mean()

# Typical use: two encoders whose outputs should agree in direction
img_enc, txt_enc = nn.Linear(512, 128), nn.Linear(300, 128)
img, txt = torch.randn(64, 512), torch.randn(64, 300)
match = torch.where(torch.rand(64) < 0.5, 1.0, -1.0)
loss = nn.CosineEmbeddingLoss(margin=0.2)(img_enc(img), txt_enc(txt), match)
```

nn.CosineEmbeddingLoss and its two-line equivalent

> **Labels are ±1, not 0/1:** `nn.CosineEmbeddingLoss` treats $y=1$ as similar and $y=-1$ as dissimilar. Passing a 0/1 label tensor silently breaks it: the $y=0$ entries match neither case and contribute **zero loss**, so every negative pair is ignored. Convert with `2 * y - 1`.

> **Near-zero vectors:** The gradient scales as $1/\lVert x\rVert$, so an embedding close to the origin gets enormous, direction-flipping updates, and at exactly zero the cosine is undefined (PyTorch's $\varepsilon$ only prevents the division by zero). Keep embeddings away from zero, for example with a normalisation layer, and watch for exploding gradients from very short vectors.

> **Choosing the margin:** $m=0$ demands only that dissimilar pairs be at least orthogonal, which in high dimensions is roughly where random vectors already sit (cos ≈ 0). Positive margins (0.1 to 0.5) make the push more selective. Negative margins push dissimilar pairs toward opposite directions, which in high dimensions is usually impossible for many classes at once and wastes capacity.

**Strengths**

- Matches cosine-similarity retrieval, so training and test measure the same thing
- Scale invariant: insensitive to embedding norms from frequency, contrast or confidence
- Bounded loss (at most 2), robust to outlier vectors with huge norms
- Built into PyTorch; same pair supervision as contrastive loss

**Limitations**

- Gradient shrinks as $1/\lVert x\rVert$ and vanishes at $\theta=0°$ and $180°$
- SGD steps lengthen unnormalised embeddings, silently lowering the effective learning rate
- Like any pair loss: one negative per term, most random negatives give zero gradient
- ±1 labels and the margin's meaning (a cosine, not a distance) are easy to get wrong

**Question:** You double the length of x₁ without changing its direction. What happens to the cosine embedding loss and its gradient with respect to x₁?

- A. The loss is unchanged and the gradient halves
- B. Both are unchanged
- C. The loss halves and the gradient is unchanged
- D. The loss is unchanged and the gradient doubles

**Answer:** A. The loss is unchanged and the gradient halves — Cosine ignores length, so $L$ is the same. Its gradient is $\frac{1}{\lVert x_1\rVert}(\hat x_2-\cos\theta\,\hat x_1)$, which halves when $\lVert x_1\rVert$ doubles (0.3536 → 0.1768 in the worked example).

**Question:** Which direction does the gradient ∂L/∂x₁ of a cosine loss point?

- A. Perpendicular to x₁
- B. Along x₁
- C. Along x₂
- D. Along x₁ − x₂

**Answer:** A. Perpendicular to x₁ — Because $\cos(cx_1,x_2)=\cos(x_1,x_2)$ for every $c>0$, the derivative in the direction of $x_1$ is zero: $x_1^\top\partial\cos/\partial x_1=0$. The gradient only rotates $x_1$.

**Key points**

- $L=1-\cos$ for $y=+1$, $\max(0,\cos-m)$ for $y=-1$: contrastive loss measured in angles (`nn.CosineEmbeddingLoss`).
- Gradient $\frac{1}{\lVert x\rVert}(\hat x_2-\cos\theta\,\hat x_1)$: perpendicular to $x$, size $\sin\theta/\lVert x\rVert$.
- Lengths are ignored by the loss but grow under SGD, which slowly shrinks the effective step: normalise or use weight decay.
- On unit vectors, $1-\cos\theta=\tfrac12\lVert\hat x_1-\hat x_2\rVert^2$, so it is the squared-distance loss on the sphere.
- Labels must be ±1; the margin is a cosine in [−1, 1], with 0 to 0.5 typical.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
