Definition
Huber loss with threshold is for residuals and beyond. The two pieces meet with the same value and the same slope at , so the loss is smooth enough for gradient descent, behaves like MSE for small errors and like MAE for large ones.
MSE and MAE each have one good half. MSE's gradient fades near the optimum, so training settles, but it grows without bound for outliers. MAE's gradient is bounded, so outliers cannot take over, but it never fades, so training jitters. Huber keeps the good half of each: the parabola where errors are small and believable, the straight line where they are large and suspicious.
Peter Huber introduced it in "Robust Estimation of a Location Parameter" (Annals of Mathematical Statistics, 1964), the paper that founded robust statistics. He asked which loss gives the best worst-case estimate of a centre when a fraction of Gaussian data may be replaced by arbitrary contamination, and the answer was this one, with set by . Deep learning rediscovered it twice: Girshick's Fast R-CNN (2015) used it under the name smooth L1 for bounding-box regression because it is less sensitive to outliers than the L2 loss of earlier detectors, and DQN (Mnih et al., 2015) clipped the temporal-difference error to , which is exactly the gradient of Huber with .
Formula and gradient
- residual; the gradient with respect to the prediction is
- threshold, in the target's units: residuals up to are treated as noise, larger ones as outliers
- the influence function: how hard one example pulls on the fit. Bounded by
- the offset inside that makes the two pieces meet: both equal at
The function is continuous with a continuous first derivative, but its second derivative jumps from 1 to 0 at . That is enough for first-order optimizers. Where a version is wanted, the pseudo-Huber loss has the same two regimes with a smooth transition.
The two limits recover the other losses. As every residual is inside the parabola and , half the squared error. As , : Huber divided by becomes MAE. That rescaled version is exactly PyTorch's smooth L1.
Properties
- Range
- ; grows like near 0 and like far away
- Gradient
- : continuous, bounded by
- Smoothness
- convex, ; curvature 1 inside and 0 outside
- Minimizer
- an M-estimator between the mean () and the median ()
- Common δ
- 1.0 on standardized targets (PyTorch default); in statistics, which keeps 95% efficiency on Gaussian data
- PyTorch
nn.HuberLoss(delta=δ)andnn.SmoothL1Loss(beta=β)
Bounded influence: outliers count as if they were only δ away
Gradient descent moves the model by the sum of the per-example gradients. Under MSE an example 8 units off pulls 8 times harder than one 1 unit off. Under Huber both pull with force at most : an outlier is still allowed to say "the fit is too low here", but not how much too low. Its pull stops growing once it is farther than , which is why sliding an outlier further away has no effect on a Huber fit.
The same fact gives a classic solver. Write with weight . Setting the gradient to zero is then a weighted least-squares problem, so Huber regression can be solved by iteratively reweighted least squares (IRLS): fit, recompute the weights from the residuals, refit. Points inside get weight 1; a point at distance gets weight 0.2.
Huber versus smooth L1 in PyTorch
PyTorch has both, and they differ only by a factor of the threshold. nn.HuberLoss(delta=δ) is above. nn.SmoothL1Loss(beta=β) is :
- same role as : where the parabola ends. Default 1.0, so the two losses coincide at their defaults
- smooth L1's linear part always has slope 1 (it is L1 with a rounded tip); Huber's has slope
- allowed: smooth L1 becomes
L1Loss.HuberLossrequires
For a fixed threshold the two give the same minimizer, but the gradient magnitudes differ by , so swapping one for the other silently changes the effective learning rate. The limits differ too. As smooth L1 goes to 0 (the parabola flattens), while Huber goes to half the MSE. As the threshold goes to 0, smooth L1 becomes L1 while Huber goes to 0.
| residual r | ℓ, δ = 1 | ∂/∂ŷ, δ = 1 | ℓ, δ = 2 | ∂/∂ŷ, δ = 2 | SmoothL1, β = 2 | ∂/∂ŷ, β = 2 |
|---|---|---|---|---|---|---|
| 0.5 | 0.125 | −0.125 | 0.125 | −0.125 | 0.0625 | −0.0625 |
| −1 | 0.5 | +0.25 | 0.5 | +0.25 | 0.25 | +0.125 |
| 2 | 1.5 | −0.25 | 2 | −0.5 | 1 | −0.25 |
| 8 | 7.5 | −0.25 | 14 | −0.5 | 7 | −0.25 |
| mean | 2.40625 | 4.15625 | 2.078125 |
Reading the table
- 1: only the first residual is inside the parabola, . The others are linear, e.g. . The mean matches the overview's , between MAE () and half the MSE ().
- 2Gradients are . With the outlier and the residual of 2 both get ; under MSE the outlier's was .
- 3: the residual of 2 sits exactly on the boundary, , and the outlier costs .
- 4Smooth L1 with is the column halved, both value and gradient. At the two losses agree to the last digit.
1import torch, torch.nn as nn23y = torch.tensor([3.0, -0.5, 2.0, 7.0])4yhat = torch.tensor([2.5, 0.5, 0.0, -1.0], requires_grad=True)56for d in (0.5, 1.0, 2.0):7 h = nn.HuberLoss(delta=d)(yhat, y).item()8 s = nn.SmoothL1Loss(beta=d)(yhat, y).item()9 print(d, h, s, h / d)10# 0.5 1.3125 2.625 2.62511# 1.0 2.40625 2.40625 2.4062512# 2.0 4.15625 2.078125 2.0781251314nn.HuberLoss(delta=2.0)(yhat, y).backward()15print(yhat.grad) # tensor([-0.1250, 0.2500, -0.5000, -0.5000]) clip(ŷ − y, ±2) / 416yhat.grad = None17nn.SmoothL1Loss(beta=2.0)(yhat, y).backward()18print(yhat.grad) # tensor([-0.0625, 0.1250, -0.2500, -0.2500]) clip((ŷ − y)/2, ±1) / 41920# Pick δ on the scale of the noise: e.g. from the residuals of a first fit21res = (y - yhat).detach()22sigma = 1.4826 * (res - res.median()).abs().median() # robust std from the MAD23delta = 1.345 * sigmaUse when
- +Bounded influence: an outlier pulls with force at most δ
- +Gradient fades near the optimum, so a fixed learning rate converges
- +Convex and C¹; easy for every optimizer
- +One knob interpolates between MSE (large δ) and MAE (small δ)
Avoid when
- −δ is an extra hyperparameter, and it is scale-dependent
- −Second derivative jumps at ±δ (use pseudo-Huber if that matters)
- −Two PyTorch variants with different gradient scales are easy to confuse
- −Still sensitive to outliers in the inputs (high leverage), not only the targets
Check yourself
Huber loss with δ = 1. One residual is 3; the network then gets worse on that example and the residual becomes 30. How does that example's gradient with respect to the prediction change?
Check yourself
A detector trained with nn.SmoothL1Loss(beta=0.1) is switched to nn.HuberLoss(delta=0.1) with everything else unchanged. What happens?
Remember
- ✓ for , beyond; gradient .
- ✓ gives half the MSE, (after dividing by ) gives MAE.
- ✓Outliers beyond δ all pull with the same capped force, so moving them further does not move the fit; near the optimum it converges like MSE.
- ✓PyTorch:
HuberLoss(delta)SmoothL1Loss(beta=delta); they coincide only at 1. - ✓Set on the scale of the noise: standardized targets and , or .