Definition
Hinge loss for a label and a real-valued score is . The product is the margin. The loss is 0 when , meaning the example is classified correctly with room to spare. Below that it grows linearly, with slope .
Cross-entropy keeps pushing every example toward more confidence, forever. Hinge loss stops once an example is "correct enough". Examples beyond the margin contribute nothing to the loss or the gradient. The solution is therefore decided by the few examples on or inside the margin: the support vectors. That is the support vector machine's loss, and it also turns up in deep learning as a classifier head, in GAN discriminators, and inside ranking and metric-learning losses.
The soft-margin SVM of Cortes and Vapnik (1995) minimizes subject to , . At the optimum each slack is , so the SVM is L2-regularized hinge-loss minimization. Two multiclass extensions followed. Weston and Watkins (1999) sum a hinge over every wrong class. Crammer and Singer (2001) charge only the worst wrong class. Tang (2013) swapped a CNN's softmax layer for a squared-hinge (L2-SVM) output and reported small gains on MNIST, CIFAR-10 and the ICML 2013 facial-expression challenge.
Formula and gradient
- the margin: positive when the sign of is right, when it is right with room to spare
- the margin target. Any positive constant works, since rescaling rescales ; regularization on is what makes "1" meaningful
- a linear score here; in a network it is the last layer's output
- the kink: any value in is a valid subgradient; autograd on
clamp(1 - m, min=0)uses 0
The gradient is a gate. Inside the margin every example pushes with the same force, , however badly it is misclassified. Outside the margin it is switched off completely. Compare the logistic loss : its gradient shrinks smoothly but never reaches 0, so every example keeps pulling forever.
Squared hinge (the L2-SVM) is differentiable everywhere, and its gradient grows with the violation. It is scikit-learn's LinearSVC default, loss="squared_hinge". The price is the same as with MSE: one badly misclassified point gets a quadratically large pull.
Properties
- Task
- binary classification with labels ; multiclass via Weston–Watkins or Crammer–Singer
- Range
- , exactly 0 for
- Convex
- yes in , so convex in for a linear model. It upper-bounds the 0–1 loss
- Smooth
- no: kink at . Squared hinge is
- Gradient
- inside the margin, 0 outside: bounded, but sparse
- Probabilities
- no. The population minimizer is with , so carries no probability information
- PyTorch
- no binary built-in:
(1 - y*f).clamp(min=0).mean(). Multiclass:nn.MultiMarginLoss. Distances:nn.HingeEmbeddingLoss
Why a margin: hinge vs logistic vs 0–1
What we care about is the 0–1 loss , but its gradient is zero almost everywhere. Hinge is a convex upper bound on it: equal to the 0–1 loss at and for every , and linear in between and below. Like the logistic loss, it is classification-calibrated (Bartlett, Jordan & McAuliffe, 2006): driving the expected surrogate risk to its minimum also drives the 0–1 risk to its minimum.
The two surrogates differ in what they leave in . Logistic loss is a negative log-likelihood, so its minimizer is the log-odds and is a probability estimate. Hinge loss is minimized by : the best hinge score is and says nothing about how sure the model is. If you need probabilities from an SVM you must calibrate afterwards (Platt scaling fits a sigmoid to the scores).
The flat region is also why SVM solutions are sparse. In the dual, every example with has coefficient , so is built only from support vectors. In a network trained with hinge loss the same thing happens per batch: correctly classified examples beyond the margin contribute nothing to that batch's gradient.
Multiclass hinge: Weston–Watkins and Crammer–Singer
With classes the model outputs a score vector and the true class should beat every other class by the margin (usually 1):
- score (logit) for class
- required gap between the true class and every rival
- on each violating , minus the number of violators on
- on the single worst rival , on , if that rival violates the margin
Weston–Watkins pushes every violator down at once. Crammer–Singer only fixes the current worst rival, which matches the multiclass 0–1 loss more closely (a mistake is a mistake, no matter how many classes beat you). In practice the two give similar accuracy. PyTorch's nn.MultiMarginLoss implements Weston–Watkins, but divides each example's sum by the number of classes .
Worked example
| margin m | hinge | ∂/∂m | squared hinge | ∂/∂m | logistic (bits) | ∂/∂m |
|---|---|---|---|---|---|---|
| (wrong) | 2.500 | −1 | 6.250 | −5.00 | 2.455 | −1.18 |
| (right, in margin) | 0.600 | −1 | 0.360 | −1.20 | 0.740 | −0.58 |
| (right, outside) | 0 | 0 | 0 | 0 | 0.114 | −0.11 |
One subgradient step of a linear SVM, J = (λ/2)‖w‖² + max(0, 1 − y·f)
- 1, , , , , learning rate . Score , margin : inside the margin, hinge loss .
- 2Gradient and (the bias is not regularized).
- 3Update: , . New score , so and the hinge loss is now exactly 0.
- 4From here on this example contributes only the shrinkage. It stops being a support vector unless shrinkage pulls it back inside the margin.
- 5Multiclass, scores for (cat, dog, car), true class cat: WW , CS .
nn.MultiMarginLossgives .
1import torch, torch.nn as nn23# Binary hinge: labels in {-1, +1}, raw scores (no sigmoid)4f = torch.tensor([-1.5, 0.4, 2.5], requires_grad=True)5y = torch.tensor([1.0, 1.0, 1.0])6loss = (1 - y * f).clamp(min=0).mean() # (2.5 + 0.6 + 0) / 3 = 1.03337loss.backward()8print(f.grad) # [-0.3333, -0.3333, 0.] <- -y/N inside the margin, 0 outside910sq = ((1 - y * f).clamp(min=0) ** 2).mean() # squared hinge (L2-SVM)1112# Multiclass (Weston-Watkins), divided by the number of classes C13s = torch.tensor([[1.0, 1.4, 0.5]])14target = torch.tensor([0]) # class index, like CrossEntropyLoss15nn.MultiMarginLoss(p=1, margin=1.0)(s, target) # 1.9 / 3 = 0.633316nn.MultiMarginLoss(p=2)(s, target) # squared terms: (1.4**2 + 0.5**2) / 3 = 0.73671718# Crammer-Singer: only the worst wrong class19def crammer_singer(s, y, margin=1.0):20 true = s.gather(1, y[:, None]).squeeze(1)21 rival = s.scatter(1, y[:, None], float("-inf")).amax(1)22 return (margin + rival - true).clamp(min=0).mean()23crammer_singer(s, target) # 1.42425# HingeEmbeddingLoss: x is a distance, y = +1 similar pair, -1 dissimilar pair26d = torch.tensor([0.3, 0.3, 1.5])27pair = torch.tensor([1.0, -1.0, -1.0])28nn.HingeEmbeddingLoss(margin=1.0)(d, pair) # (0.3 + 0.7 + 0) / 3 = 0.3333Use when
- +Exactly zero loss and gradient beyond the margin: learning focuses on hard and borderline examples
- +Bounded gradient () inside the margin: robust to badly mislabelled points, unlike squared hinge or MSE
- +Convex for linear models, with the max-margin interpretation and sparse support-vector solutions
- +Natural building block for ranking, metric learning and GAN discriminators
Avoid when
- −Scores are not probabilities; you need Platt scaling or similar to calibrate them
- −Not differentiable at (squared hinge fixes this but loses outlier robustness)
- −Sparse gradients: once most examples clear the margin, each batch carries little signal
- −Margin is only meaningful together with a weight-norm constraint
Check yourself
A linear SVM example has y = +1 and f(x) = 1.7. What does it contribute to ∂(hinge)/∂w?
Check yourself
Scores (cat, dog, car) = (2.0, 2.5, 1.8), true class cat, Δ = 1. What are the Weston–Watkins and Crammer–Singer losses?
Remember
- ✓ with : zero for margin , slope below it.
- ✓Only margin violators (support vectors) get gradient; logistic loss, by contrast, pulls every example forever.
- ✓Soft-margin SVM L2-regularized hinge minimization; the norm penalty is what gives the margin "1" its meaning.
- ✓Squared hinge is smooth (scikit-learn's
LinearSVCdefault) but quadratically sensitive to outliers. - ✓Multiclass: Weston–Watkins sums over violating rivals (
nn.MultiMarginLoss, divided by ); Crammer–Singer charges only the worst. - ✓Hinge scores are not probabilities: calibrate them if you need confidence estimates.