# Hinge loss

> max(0, 1 − y·f(x)): zero once an example is on the right side with margin 1, linear otherwise. It is the SVM's loss. Only margin violators get gradient, and the scores it learns are not probabilities.

Source: https://www.mlfoundry.in/learn/hinge-loss  
Section: Deep Learning › Loss Functions  
Level: core · 10 min read · updated 2026-10-11

**Hinge loss:** for a label $y\in\{-1,+1\}$ and a real-valued score $f(x)$ is $\ell(y,f)=\max\big(0,\,1-y f(x)\big)$. The product $m=y f(x)$ is the **margin**. The loss is 0 when $m\ge1$, meaning the example is classified correctly with room to spare. Below that it grows linearly, with slope $-1$.

Cross-entropy keeps pushing every example toward more confidence, forever. Hinge loss stops once an example is "correct enough". Examples beyond the margin contribute nothing to the loss or the gradient. The solution is therefore decided by the few examples on or inside the margin: the **support vectors**. That is the support vector machine's loss, and it also turns up in deep learning as a classifier head, in GAN discriminators, and inside ranking and metric-learning losses.

The soft-margin SVM of Cortes and Vapnik (1995) minimizes $\tfrac12\|w\|^2+C\sum_i\xi_i$ subject to $y_i(w^\top x_i+b)\ge1-\xi_i$, $\xi_i\ge0$. At the optimum each slack is $\xi_i=\max(0,1-y_if(x_i))$, so the SVM is L2-regularized hinge-loss minimization. Two multiclass extensions followed. Weston and Watkins (1999) sum a hinge over every wrong class. Crammer and Singer (2001) charge only the worst wrong class. Tang (2013) swapped a CNN's softmax layer for a squared-hinge (L2-SVM) output and reported small gains on MNIST, CIFAR-10 and the ICML 2013 facial-expression challenge.

## Formula and gradient

$$
\ell_{\text{hinge}}(m)=\max(0,\,1-m),\qquad \frac{\partial\ell}{\partial m}=\begin{cases}-1 & m<1\\ 0 & m>1\end{cases},\qquad \frac{\partial\ell}{\partial w}=\begin{cases}-y\,x & m<1\\ 0 & m>1\end{cases}
$$

- $m=y\,f(x)$: the margin: positive when the sign of $f$ is right, $\ge1$ when it is right with room to spare
- $1$: the margin target. Any positive constant works, since rescaling $w$ rescales $f$; regularization on $\|w\|$ is what makes "1" meaningful
- $f(x)=w^\top x+b$: a linear score here; in a network it is the last layer's output
- $m=1$: the kink: any value in $[-1,0]$ is a valid subgradient; autograd on `clamp(1 - m, min=0)` uses 0

The gradient is a gate. Inside the margin every example pushes with the same force, $-y\,x$, however badly it is misclassified. Outside the margin it is switched off completely. Compare the logistic loss $\log(1+e^{-m})$: its gradient $-\sigma(-m)$ shrinks smoothly but never reaches 0, so every example keeps pulling forever.

**Squared hinge** $\max(0,1-m)^2$ (the L2-SVM) is differentiable everywhere, and its gradient $-2\max(0,1-m)$ grows with the violation. It is scikit-learn's `LinearSVC` default, `loss="squared_hinge"`. The price is the same as with MSE: one badly misclassified point gets a quadratically large pull.

**Properties**

- **Task:** binary classification with labels $\pm1$; multiclass via Weston–Watkins or Crammer–Singer
- **Range:** $[0,\infty)$, exactly 0 for $m\ge1$
- **Convex:** yes in $m$, so convex in $w$ for a linear model. It upper-bounds the 0–1 loss
- **Smooth:** no: kink at $m=1$. Squared hinge is $C^1$
- **Gradient:** $-1$ inside the margin, 0 outside: bounded, but sparse
- **Probabilities:** no. The population minimizer is $\operatorname{sign}(2\eta(x)-1)$ with $\eta=P(y=1\mid x)$, so $f$ carries no probability information
- **PyTorch:** no binary built-in: `(1 - y*f).clamp(min=0).mean()`. Multiclass: `nn.MultiMarginLoss`. Distances: `nn.HingeEmbeddingLoss`

> Figure: Left: hinge, squared hinge, logistic loss in bits and the 0–1 loss against the margin m. All three surrogates pass through 1 at m = 0. At the default m = 0.40 the hinge loss is 0.600 with gradient −1.00, squared hinge 0.360 with −1.20, and logistic 0.740 with −0.58. Move m past 1 and hinge and squared hinge drop to exactly 0 with zero gradient, while logistic keeps a pull (−0.11 at m = 2.5). Right: three class scores with true class cat. At the default scores (1.0, 1.4, 0.5), dog and car both sit right of the s_y − Δ line. Weston–Watkins charges both (1.4 + 0.5 = 1.90, gradient (−2, +1, +1)). Crammer–Singer charges only dog (1.40, gradient (−1, +1, 0)). PyTorch's MultiMarginLoss reports 1.90 / 3 = 0.6333.

## Why a margin: hinge vs logistic vs 0–1

What we care about is the 0–1 loss $\mathbb 1[m\le0]$, but its gradient is zero almost everywhere. Hinge is a convex upper bound on it: equal to the 0–1 loss at $m=0$ and for every $m\ge1$, and linear in between and below. Like the logistic loss, it is **classification-calibrated** (Bartlett, Jordan & McAuliffe, 2006): driving the expected surrogate risk to its minimum also drives the 0–1 risk to its minimum.

The two surrogates differ in what they leave in $f$. Logistic loss is a negative log-likelihood, so its minimizer is the log-odds and $\sigma(f)$ is a probability estimate. Hinge loss is minimized by $\operatorname{sign}(2\eta-1)$: the best hinge score is $\pm1$ and says nothing about how sure the model is. If you need probabilities from an SVM you must calibrate afterwards (Platt scaling fits a sigmoid to the scores).

The flat region is also why SVM solutions are sparse. In the dual, every example with $m_i>1$ has coefficient $\alpha_i=0$, so $w=\sum_i\alpha_iy_ix_i$ is built only from support vectors. In a network trained with hinge loss the same thing happens per batch: correctly classified examples beyond the margin contribute nothing to that batch's gradient.

## Multiclass hinge: Weston–Watkins and Crammer–Singer

With $C$ classes the model outputs a score vector $s\in\mathbb R^C$ and the true class $y$ should beat every other class by the margin $\Delta$ (usually 1):

$$
\ell_{\text{WW}}=\sum_{j\ne y}\max\big(0,\,\Delta+s_j-s_y\big)\qquad\qquad \ell_{\text{CS}}=\max\Big(0,\,\Delta+\max_{j\ne y}s_j-s_y\Big)
$$

- $s_j$: score (logit) for class $j$
- $\Delta$: required gap between the true class and every rival
- $\partial\ell_{\text{WW}}/\partial s$: $+1$ on each violating $j$, minus the number of violators on $s_y$
- $\partial\ell_{\text{CS}}/\partial s$: $+1$ on the single worst rival $j^\star$, $-1$ on $s_y$, if that rival violates the margin

Weston–Watkins pushes every violator down at once. Crammer–Singer only fixes the current worst rival, which matches the multiclass 0–1 loss more closely (a mistake is a mistake, no matter how many classes beat you). In practice the two give similar accuracy. PyTorch's `nn.MultiMarginLoss` implements Weston–Watkins, but divides each example's sum by the number of classes $C$.

## Worked example

Binary surrogates at three margins (logistic loss in bits, log₂(1 + e^(−m)); values from the figure's lib)

| margin m | hinge | ∂/∂m | squared hinge | ∂/∂m | logistic (bits) | ∂/∂m |
| --- | --- | --- | --- | --- | --- | --- |
| $-1.5$ (wrong) | 2.500 | −1 | 6.250 | −5.00 | 2.455 | −1.18 |
| $0.4$ (right, in margin) | 0.600 | −1 | 0.360 | −1.20 | 0.740 | −0.58 |
| $2.5$ (right, outside) | 0 | 0 | 0 | 0 | 0.114 | −0.11 |

**One subgradient step of a linear SVM, J = (λ/2)‖w‖² + max(0, 1 − y·f)**

1. $x=(2,1)$, $y=+1$, $w=(0.5,-0.4)$, $b=0$, $\lambda=0.1$, learning rate $0.1$. Score $f=1.0-0.4=0.6$, margin $m=0.6<1$: inside the margin, hinge loss $0.4$.
2. Gradient $\partial J/\partial w=\lambda w-yx=(0.05-2,\,-0.04-1)=(-1.95,\,-1.04)$ and $\partial J/\partial b=-y=-1$ (the bias is not regularized).
3. Update: $w\leftarrow(0.695,\,-0.296)$, $b\leftarrow0.1$. New score $f=1.39-0.296+0.1=1.194$, so $m>1$ and the hinge loss is now exactly 0.
4. From here on this example contributes only the $\lambda w$ shrinkage. It stops being a support vector unless shrinkage pulls it back inside the margin.
5. Multiclass, scores $s=(1.0,\,1.4,\,0.5)$ for (cat, dog, car), true class cat: WW $=\max(0,1.4)+\max(0,0.5)=1.9$, CS $=1.4$. `nn.MultiMarginLoss` gives $1.9/3=0.6333$.

```python
import torch, torch.nn as nn

# Binary hinge: labels in {-1, +1}, raw scores (no sigmoid)
f = torch.tensor([-1.5, 0.4, 2.5], requires_grad=True)
y = torch.tensor([1.0, 1.0, 1.0])
loss = (1 - y * f).clamp(min=0).mean()          # (2.5 + 0.6 + 0) / 3 = 1.0333
loss.backward()
print(f.grad)                                    # [-0.3333, -0.3333, 0.]  <- -y/N inside the margin, 0 outside

sq = ((1 - y * f).clamp(min=0) ** 2).mean()      # squared hinge (L2-SVM)

# Multiclass (Weston-Watkins), divided by the number of classes C
s = torch.tensor([[1.0, 1.4, 0.5]])
target = torch.tensor([0])                       # class index, like CrossEntropyLoss
nn.MultiMarginLoss(p=1, margin=1.0)(s, target)   # 1.9 / 3 = 0.6333
nn.MultiMarginLoss(p=2)(s, target)               # squared terms: (1.4**2 + 0.5**2) / 3 = 0.7367

# Crammer-Singer: only the worst wrong class
def crammer_singer(s, y, margin=1.0):
    true = s.gather(1, y[:, None]).squeeze(1)
    rival = s.scatter(1, y[:, None], float("-inf")).amax(1)
    return (margin + rival - true).clamp(min=0).mean()
crammer_singer(s, target)                        # 1.4

# HingeEmbeddingLoss: x is a distance, y = +1 similar pair, -1 dissimilar pair
d = torch.tensor([0.3, 0.3, 1.5])
pair = torch.tensor([1.0, -1.0, -1.0])
nn.HingeEmbeddingLoss(margin=1.0)(d, pair)       # (0.3 + 0.7 + 0) / 3 = 0.3333
```

Binary hinge by hand, PyTorch's multiclass and embedding variants

> **Labels must be ±1, scores must be raw:** Hinge loss assumes $y\in\{-1,+1\}$. Feeding $\{0,1\}$ labels makes every negative example have $m=0$ and loss 1 whatever the score, so they contribute a constant and no gradient. Do not squash $f$ through a sigmoid either: with $f\in(0,1)$ a positive example can never reach $m\ge1$, so it always sits in the margin.

> **Hinge needs a norm constraint:** Without regularization the margin "1" is meaningless: doubling $w$ doubles every margin, so any separating $w$ can be scaled until the loss is zero, and training just grows the weights. The SVM fixes the scale with $\tfrac\lambda2\|w\|^2$. A network needs weight decay or normalization for the margin to mean anything.

> **The same hinge, everywhere:** Many losses are a hinge on a difference of scores. `nn.MarginRankingLoss` is $\max(0,-y(x_1-x_2)+\text{margin})$. Triplet loss is $\max(0,\,d(a,p)-d(a,n)+\alpha)$. The GAN hinge loss used in SAGAN and BigGAN trains the discriminator with $\max(0,1-D(x))+\max(0,1+D(G(z)))$. Each one stops caring once the gap is big enough.

> **Relation to cross-entropy:** Logistic loss $\log(1+e^{-m})$ is a smoothed hinge. Softmax cross-entropy is a smoothed Crammer–Singer: $\log\sum_je^{s_j}-s_y$ is a soft maximum over rivals minus the true score. That is why both families train to similar accuracy, and cross-entropy wins whenever you need calibrated probabilities.

**Strengths**

- Exactly zero loss and gradient beyond the margin: learning focuses on hard and borderline examples
- Bounded gradient ($-1$) inside the margin: robust to badly mislabelled points, unlike squared hinge or MSE
- Convex for linear models, with the max-margin interpretation and sparse support-vector solutions
- Natural building block for ranking, metric learning and GAN discriminators

**Limitations**

- Scores are not probabilities; you need Platt scaling or similar to calibrate them
- Not differentiable at $m=1$ (squared hinge fixes this but loses outlier robustness)
- Sparse gradients: once most examples clear the margin, each batch carries little signal
- Margin is only meaningful together with a weight-norm constraint

**Question:** A linear SVM example has y = +1 and f(x) = 1.7. What does it contribute to ∂(hinge)/∂w?

- A. Nothing: its margin is above 1
- B. −x, since it is a positive example
- C. −0.7·x
- D. A small pull proportional to e^(−1.7)

**Answer:** A. Nothing: its margin is above 1 — $m=yf=1.7>1$, so $\max(0,1-m)=0$ on a neighbourhood of this $w$ and the gradient is 0. Only examples with $m<1$ (the support vectors) shape the solution. The fourth option describes the logistic loss, whose gradient never vanishes.

**Question:** Scores (cat, dog, car) = (2.0, 2.5, 1.8), true class cat, Δ = 1. What are the Weston–Watkins and Crammer–Singer losses?

- A. WW 2.3, CS 1.5
- B. WW 1.5, CS 1.5
- C. WW 2.3, CS 0.8
- D. WW 0.77, CS 1.5

**Answer:** A. WW 2.3, CS 1.5 — Dog term $1+2.5-2.0=1.5$, car term $1+1.8-2.0=0.8$. WW sums them: $2.3$. CS takes only the worst rival, dog: $1.5$. ($2.3/3=0.77$ is what `nn.MultiMarginLoss` would print.)

**Key points**

- $\ell=\max(0,1-yf)$ with $y\in\{\pm1\}$: zero for margin $m\ge1$, slope $-1$ below it.
- Only margin violators (support vectors) get gradient; logistic loss, by contrast, pulls every example forever.
- Soft-margin SVM $=$ L2-regularized hinge minimization; the norm penalty is what gives the margin "1" its meaning.
- Squared hinge is smooth (scikit-learn's `LinearSVC` default) but quadratically sensitive to outliers.
- Multiclass: Weston–Watkins sums over violating rivals (`nn.MultiMarginLoss`, divided by $C$); Crammer–Singer charges only the worst.
- Hinge scores are not probabilities: calibrate them if you need confidence estimates.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
