Hinge loss

max(0, 1 − y·f(x)): zero once an example is on the right side with margin 1, linear otherwise. It is the SVM's loss. Only margin violators get gradient, and the scores it learns are not probabilities.

10 min read914 wordscore0 viewsupdated 2026-10-11

Definition

Hinge loss for a label y∈{−1,+1}y\in\{-1,+1\} and a real-valued score f(x)f(x) is ℓ(y,f)=max⁡(0, 1−yf(x))\ell(y,f)=\max\big(0,\,1-y f(x)\big). The product m=yf(x)m=y f(x) is the margin. The loss is 0 when m≥1m\ge1, meaning the example is classified correctly with room to spare. Below that it grows linearly, with slope −1-1.

Cross-entropy keeps pushing every example toward more confidence, forever. Hinge loss stops once an example is "correct enough". Examples beyond the margin contribute nothing to the loss or the gradient. The solution is therefore decided by the few examples on or inside the margin: the support vectors. That is the support vector machine's loss, and it also turns up in deep learning as a classifier head, in GAN discriminators, and inside ranking and metric-learning losses.

The soft-margin SVM of Cortes and Vapnik (1995) minimizes 12∥w∥2+C∑iξi\tfrac12\|w\|^2+C\sum_i\xi_i subject to yi(w⊤xi+b)≥1−ξiy_i(w^\top x_i+b)\ge1-\xi_i, ξi≥0\xi_i\ge0. At the optimum each slack is ξi=max⁡(0,1−yif(xi))\xi_i=\max(0,1-y_if(x_i)), so the SVM is L2-regularized hinge-loss minimization. Two multiclass extensions followed. Weston and Watkins (1999) sum a hinge over every wrong class. Crammer and Singer (2001) charge only the worst wrong class. Tang (2013) swapped a CNN's softmax layer for a squared-hinge (L2-SVM) output and reported small gains on MNIST, CIFAR-10 and the ICML 2013 facial-expression challenge.

Formula and gradient

ℓhinge(m)=max⁡(0, 1−m),∂ℓ∂m={−1m<10m>1,∂ℓ∂w={−y xm<10m>1\ell_{\text{hinge}}(m)=\max(0,\,1-m),\qquad \frac{\partial\ell}{\partial m}=\begin{cases}-1 & m<1\\ 0 & m>1\end{cases},\qquad \frac{\partial\ell}{\partial w}=\begin{cases}-y\,x & m<1\\ 0 & m>1\end{cases}
m=y f(x)m=y\,f(x)
the margin: positive when the sign of ff is right, ≥1\ge1 when it is right with room to spare
11
the margin target. Any positive constant works, since rescaling ww rescales ff; regularization on ∥w∥\|w\| is what makes "1" meaningful
f(x)=w⊤x+bf(x)=w^\top x+b
a linear score here; in a network it is the last layer's output
m=1m=1
the kink: any value in [−1,0][-1,0] is a valid subgradient; autograd on clamp(1 - m, min=0) uses 0

The gradient is a gate. Inside the margin every example pushes with the same force, −y x-y\,x, however badly it is misclassified. Outside the margin it is switched off completely. Compare the logistic loss log⁡(1+e−m)\log(1+e^{-m}): its gradient −σ(−m)-\sigma(-m) shrinks smoothly but never reaches 0, so every example keeps pulling forever.

Squared hinge max⁡(0,1−m)2\max(0,1-m)^2 (the L2-SVM) is differentiable everywhere, and its gradient −2max⁡(0,1−m)-2\max(0,1-m) grows with the violation. It is scikit-learn's LinearSVC default, loss="squared_hinge". The price is the same as with MSE: one badly misclassified point gets a quadratically large pull.

Properties

Task
binary classification with labels ±1\pm1; multiclass via Weston–Watkins or Crammer–Singer
Range
[0,∞)[0,\infty), exactly 0 for m≥1m\ge1
Convex
yes in mm, so convex in ww for a linear model. It upper-bounds the 0–1 loss
Smooth
no: kink at m=1m=1. Squared hinge is C1C^1
Gradient
−1-1 inside the margin, 0 outside: bounded, but sparse
Probabilities
no. The population minimizer is sign⁡(2η(x)−1)\operatorname{sign}(2\eta(x)-1) with η=P(y=1∣x)\eta=P(y=1\mid x), so ff carries no probability information
PyTorch
no binary built-in: (1 - y*f).clamp(min=0).mean(). Multiclass: nn.MultiMarginLoss. Distances: nn.HingeEmbeddingLoss
binary: ℓ(m), m = y·f(x)inside margin01234−2−10123lossℓ(m)dℓ/dmhinge0.600−1.00squared hinge0.360−1.20logistic (bits)0.740−0.580–10.0000 (a.e.)correct, in marginhinge pushes:∂ℓ/∂w = −y·xmulticlass, Δ = 1, true ★ catwrong class j violates if sⱼ > s_y − Δs_y − Δ−3−2−10123cat ★ s=1.00dog s=1.40term 1.40car s=0.50term 0.50loss∂L/∂sWeston–Watkins1.90−2 +1 +1Crammer–Singer1.40−1 +1 0nn.MultiMarginLoss = WW / C = 0.6333CS charges only the worst violator
highlight
true class
fig.Left: hinge, squared hinge, logistic loss in bits and the 0–1 loss against the margin m. All three surrogates pass through 1 at m = 0. At the default m = 0.40 the hinge loss is 0.600 with gradient −1.00, squared hinge 0.360 with −1.20, and logistic 0.740 with −0.58. Move m past 1 and hinge and squared hinge drop to exactly 0 with zero gradient, while logistic keeps a pull (−0.11 at m = 2.5). Right: three class scores with true class cat. At the default scores (1.0, 1.4, 0.5), dog and car both sit right of the s_y − Δ line. Weston–Watkins charges both (1.4 + 0.5 = 1.90, gradient (−2, +1, +1)). Crammer–Singer charges only dog (1.40, gradient (−1, +1, 0)). PyTorch's MultiMarginLoss reports 1.90 / 3 = 0.6333.

Why a margin: hinge vs logistic vs 0–1

What we care about is the 0–1 loss 1[m≤0]\mathbb 1[m\le0], but its gradient is zero almost everywhere. Hinge is a convex upper bound on it: equal to the 0–1 loss at m=0m=0 and for every m≥1m\ge1, and linear in between and below. Like the logistic loss, it is classification-calibrated (Bartlett, Jordan & McAuliffe, 2006): driving the expected surrogate risk to its minimum also drives the 0–1 risk to its minimum.

The two surrogates differ in what they leave in ff. Logistic loss is a negative log-likelihood, so its minimizer is the log-odds and σ(f)\sigma(f) is a probability estimate. Hinge loss is minimized by sign⁡(2η−1)\operatorname{sign}(2\eta-1): the best hinge score is ±1\pm1 and says nothing about how sure the model is. If you need probabilities from an SVM you must calibrate afterwards (Platt scaling fits a sigmoid to the scores).

The flat region is also why SVM solutions are sparse. In the dual, every example with mi>1m_i>1 has coefficient αi=0\alpha_i=0, so w=∑iαiyixiw=\sum_i\alpha_iy_ix_i is built only from support vectors. In a network trained with hinge loss the same thing happens per batch: correctly classified examples beyond the margin contribute nothing to that batch's gradient.

Multiclass hinge: Weston–Watkins and Crammer–Singer

With CC classes the model outputs a score vector s∈RCs\in\mathbb R^C and the true class yy should beat every other class by the margin Δ\Delta (usually 1):

ℓWW=∑j≠ymax⁡(0, Δ+sj−sy)ℓCS=max⁡(0, Δ+max⁡j≠ysj−sy)\ell_{\text{WW}}=\sum_{j\ne y}\max\big(0,\,\Delta+s_j-s_y\big)\qquad\qquad \ell_{\text{CS}}=\max\Big(0,\,\Delta+\max_{j\ne y}s_j-s_y\Big)
sjs_j
score (logit) for class jj
Δ\Delta
required gap between the true class and every rival
∂ℓWW/∂s\partial\ell_{\text{WW}}/\partial s
+1+1 on each violating jj, minus the number of violators on sys_y
∂ℓCS/∂s\partial\ell_{\text{CS}}/\partial s
+1+1 on the single worst rival j⋆j^\star, −1-1 on sys_y, if that rival violates the margin

Weston–Watkins pushes every violator down at once. Crammer–Singer only fixes the current worst rival, which matches the multiclass 0–1 loss more closely (a mistake is a mistake, no matter how many classes beat you). In practice the two give similar accuracy. PyTorch's nn.MultiMarginLoss implements Weston–Watkins, but divides each example's sum by the number of classes CC.

Worked example

margin mhinge∂/∂msquared hinge∂/∂mlogistic (bits)∂/∂m
−1.5-1.5 (wrong)2.500−16.250−5.002.455−1.18
0.40.4 (right, in margin)0.600−10.360−1.200.740−0.58
2.52.5 (right, outside)00000.114−0.11
Binary surrogates at three margins (logistic loss in bits, log₂(1 + e^(−m)); values from the figure's lib)

One subgradient step of a linear SVM, J = (λ/2)‖w‖² + max(0, 1 − y·f)

  1. 1x=(2,1)x=(2,1), y=+1y=+1, w=(0.5,−0.4)w=(0.5,-0.4), b=0b=0, λ=0.1\lambda=0.1, learning rate 0.10.1. Score f=1.0−0.4=0.6f=1.0-0.4=0.6, margin m=0.6<1m=0.6<1: inside the margin, hinge loss 0.40.4.
  2. 2Gradient ∂J/∂w=λw−yx=(0.05−2, −0.04−1)=(−1.95, −1.04)\partial J/\partial w=\lambda w-yx=(0.05-2,\,-0.04-1)=(-1.95,\,-1.04) and ∂J/∂b=−y=−1\partial J/\partial b=-y=-1 (the bias is not regularized).
  3. 3Update: w←(0.695, −0.296)w\leftarrow(0.695,\,-0.296), b←0.1b\leftarrow0.1. New score f=1.39−0.296+0.1=1.194f=1.39-0.296+0.1=1.194, so m>1m>1 and the hinge loss is now exactly 0.
  4. 4From here on this example contributes only the λw\lambda w shrinkage. It stops being a support vector unless shrinkage pulls it back inside the margin.
  5. 5Multiclass, scores s=(1.0, 1.4, 0.5)s=(1.0,\,1.4,\,0.5) for (cat, dog, car), true class cat: WW =max⁡(0,1.4)+max⁡(0,0.5)=1.9=\max(0,1.4)+\max(0,0.5)=1.9, CS =1.4=1.4. nn.MultiMarginLoss gives 1.9/3=0.63331.9/3=0.6333.
python
1import torch, torch.nn as nn
2
3# Binary hinge: labels in {-1, +1}, raw scores (no sigmoid)
4f = torch.tensor([-1.5, 0.4, 2.5], requires_grad=True)
5y = torch.tensor([1.0, 1.0, 1.0])
6loss = (1 - y * f).clamp(min=0).mean() # (2.5 + 0.6 + 0) / 3 = 1.0333
7loss.backward()
8print(f.grad) # [-0.3333, -0.3333, 0.] <- -y/N inside the margin, 0 outside
9
10sq = ((1 - y * f).clamp(min=0) ** 2).mean() # squared hinge (L2-SVM)
11
12# Multiclass (Weston-Watkins), divided by the number of classes C
13s = torch.tensor([[1.0, 1.4, 0.5]])
14target = torch.tensor([0]) # class index, like CrossEntropyLoss
15nn.MultiMarginLoss(p=1, margin=1.0)(s, target) # 1.9 / 3 = 0.6333
16nn.MultiMarginLoss(p=2)(s, target) # squared terms: (1.4**2 + 0.5**2) / 3 = 0.7367
17
18# Crammer-Singer: only the worst wrong class
19def crammer_singer(s, y, margin=1.0):
20 true = s.gather(1, y[:, None]).squeeze(1)
21 rival = s.scatter(1, y[:, None], float("-inf")).amax(1)
22 return (margin + rival - true).clamp(min=0).mean()
23crammer_singer(s, target) # 1.4
24
25# HingeEmbeddingLoss: x is a distance, y = +1 similar pair, -1 dissimilar pair
26d = torch.tensor([0.3, 0.3, 1.5])
27pair = torch.tensor([1.0, -1.0, -1.0])
28nn.HingeEmbeddingLoss(margin=1.0)(d, pair) # (0.3 + 0.7 + 0) / 3 = 0.3333
Binary hinge by hand, PyTorch's multiclass and embedding variants

Use when

  • +Exactly zero loss and gradient beyond the margin: learning focuses on hard and borderline examples
  • +Bounded gradient (−1-1) inside the margin: robust to badly mislabelled points, unlike squared hinge or MSE
  • +Convex for linear models, with the max-margin interpretation and sparse support-vector solutions
  • +Natural building block for ranking, metric learning and GAN discriminators

Avoid when

  • −Scores are not probabilities; you need Platt scaling or similar to calibrate them
  • −Not differentiable at m=1m=1 (squared hinge fixes this but loses outlier robustness)
  • −Sparse gradients: once most examples clear the margin, each batch carries little signal
  • −Margin is only meaningful together with a weight-norm constraint

Check yourself

A linear SVM example has y = +1 and f(x) = 1.7. What does it contribute to ∂(hinge)/∂w?

Check yourself

Scores (cat, dog, car) = (2.0, 2.5, 1.8), true class cat, Δ = 1. What are the Weston–Watkins and Crammer–Singer losses?

Remember

  • ✓ℓ=max⁡(0,1−yf)\ell=\max(0,1-yf) with y∈{±1}y\in\{\pm1\}: zero for margin m≥1m\ge1, slope −1-1 below it.
  • ✓Only margin violators (support vectors) get gradient; logistic loss, by contrast, pulls every example forever.
  • ✓Soft-margin SVM == L2-regularized hinge minimization; the norm penalty is what gives the margin "1" its meaning.
  • ✓Squared hinge is smooth (scikit-learn's LinearSVC default) but quadratically sensitive to outliers.
  • ✓Multiclass: Weston–Watkins sums over violating rivals (nn.MultiMarginLoss, divided by CC); Crammer–Singer charges only the worst.
  • ✓Hinge scores are not probabilities: calibrate them if you need confidence estimates.
← back to Deep Learning roadmap