Huber loss

Quadratic for residuals up to δ, linear beyond, so its gradient is the residual clipped to ±δ: MSE-like smooth convergence near the target, MAE-like bounded influence for outliers. PyTorch's SmoothL1Loss is the same curve divided by β.

10 min read865 wordscore0 viewsupdated 2026-10-11

Definition

Huber loss with threshold δ>0\delta>0 is 12r2\tfrac12 r^2 for residuals ∣r∣≤δ|r|\le\delta and δ(∣r∣−12δ)\delta(|r|-\tfrac12\delta) beyond. The two pieces meet with the same value and the same slope at ∣r∣=δ|r|=\delta, so the loss is smooth enough for gradient descent, behaves like MSE for small errors and like MAE for large ones.

MSE and MAE each have one good half. MSE's gradient fades near the optimum, so training settles, but it grows without bound for outliers. MAE's gradient is bounded, so outliers cannot take over, but it never fades, so training jitters. Huber keeps the good half of each: the parabola where errors are small and believable, the straight line where they are large and suspicious.

Peter Huber introduced it in "Robust Estimation of a Location Parameter" (Annals of Mathematical Statistics, 1964), the paper that founded robust statistics. He asked which loss gives the best worst-case estimate of a centre when a fraction ε\varepsilon of Gaussian data may be replaced by arbitrary contamination, and the answer was this one, with δ\delta set by ε\varepsilon. Deep learning rediscovered it twice: Girshick's Fast R-CNN (2015) used it under the name smooth L1 for bounding-box regression because it is less sensitive to outliers than the L2 loss of earlier detectors, and DQN (Mnih et al., 2015) clipped the temporal-difference error to [−1,1][-1,1], which is exactly the gradient of Huber with δ=1\delta=1.

Formula and gradient

ℓδ(r)={12r2∣r∣≤δδ(∣r∣−12δ)∣r∣>δψδ(r)=dℓδdr=clip⁡(r,−δ,δ)={r∣r∣≤δδsign⁡(r)∣r∣>δ\ell_\delta(r)=\begin{cases}\tfrac12 r^2 & |r|\le\delta\\[2pt] \delta\big(|r|-\tfrac12\delta\big) & |r|>\delta\end{cases}\qquad\qquad \psi_\delta(r)=\frac{d\ell_\delta}{dr}=\operatorname{clip}(r,-\delta,\delta)=\begin{cases}r & |r|\le\delta\\ \delta\operatorname{sign}(r) & |r|>\delta\end{cases}
r=y−y^r=y-\hat y
residual; the gradient with respect to the prediction is ∂ℓ/∂y^=−ψδ(r)=clip⁡(y^−y,−δ,δ)\partial\ell/\partial\hat y=-\psi_\delta(r)=\operatorname{clip}(\hat y-y,-\delta,\delta)
δ\delta
threshold, in the target's units: residuals up to δ\delta are treated as noise, larger ones as outliers
ψδ\psi_\delta
the influence function: how hard one example pulls on the fit. Bounded by δ\delta
−12δ2-\tfrac12\delta^2
the offset inside δ(∣r∣−12δ)\delta(|r|-\tfrac12\delta) that makes the two pieces meet: both equal 12δ2\tfrac12\delta^2 at ∣r∣=δ|r|=\delta

The function is continuous with a continuous first derivative, but its second derivative jumps from 1 to 0 at ∣r∣=δ|r|=\delta. That is enough for first-order optimizers. Where a C∞C^\infty version is wanted, the pseudo-Huber loss δ2(1+(r/δ)2−1)\delta^2\big(\sqrt{1+(r/\delta)^2}-1\big) has the same two regimes with a smooth transition.

The two limits recover the other losses. As δ→∞\delta\to\infty every residual is inside the parabola and ℓδ=12r2\ell_\delta=\tfrac12r^2, half the squared error. As δ→0\delta\to0, ℓδ/δ→∣r∣\ell_\delta/\delta\to|r|: Huber divided by δ\delta becomes MAE. That rescaled version is exactly PyTorch's smooth L1.

Properties

Range
[0,∞)[0,\infty); grows like 12r2\tfrac12r^2 near 0 and like δ∣r∣\delta|r| far away
Gradient
clip⁡(y^−y,−δ,δ)\operatorname{clip}(\hat y-y,-\delta,\delta): continuous, bounded by δ\delta
Smoothness
convex, C1C^1; curvature 1 inside ±δ\pm\delta and 0 outside
Minimizer
an M-estimator between the mean (δ→∞\delta\to\infty) and the median (δ→0\delta\to0)
Common δ
1.0 on standardized targets (PyTorch default); 1.345σ^1.345\hat\sigma in statistics, which keeps 95% efficiency on Gaussian data
PyTorch
nn.HuberLoss(delta=δ) and nn.SmoothL1Loss(beta=β) =ℓβ/β=\ell_\beta/\beta

Bounded influence: outliers count as if they were only δ away

Gradient descent moves the model by the sum of the per-example gradients. Under MSE an example 8 units off pulls 8 times harder than one 1 unit off. Under Huber both pull with force at most δ\delta: an outlier is still allowed to say "the fit is too low here", but not how much too low. Its pull stops growing once it is farther than δ\delta, which is why sliding an outlier further away has no effect on a Huber fit.

The same fact gives a classic solver. Write ψδ(r)=w(r) r\psi_\delta(r)=w(r)\,r with weight w(r)=min⁡(1,δ/∣r∣)w(r)=\min(1,\delta/|r|). Setting the gradient to zero is then a weighted least-squares problem, so Huber regression can be solved by iteratively reweighted least squares (IRLS): fit, recompute the weights from the residuals, refit. Points inside ±δ\pm\delta get weight 1; a point at distance 5δ5\delta gets weight 0.2.

true a = 1, b = 0.5; dot opacity = Huber weight−10−5050246810∂Huber/∂r = clip(r, ±δ)−4−2024−10−50510residual r = y − ŷ (Huber fit)━ MSE fit a = 1.90, b = 0.160 · ━ Huber δ=1.00 a = 1.31, b = 0.4042 of 14 residuals beyond δ (gradient capped) · ┄ ψ = r: MSE's unbounded gradientat the Huber fit: HuberLoss(δ) = 0.7896 · SmoothL1Loss(β=δ) = 0.7896 = HuberLoss/δ
Huber fit by IRLS, wᵢ = min(1, δ/|rᵢ|)
fig.Fourteen points around the line y = 1 + 0.5x (noise σ = 0.3), two of them pulled down by 6. Least squares (blue) is dragged to slope 0.160; Huber with δ = 1 (green, solved by IRLS) keeps 0.404, and the two outliers get weights 0.19 and 0.18 (dot opacity). Pull them down to 10: the least-squares slope falls to −0.058 while the Huber line does not move at all, because both outliers sit on the flat part of the clipped gradient (right). Shrink δ to 0.4 (about 1.345σ) and the slope recovers to 0.452; raise δ past 4 and no residual is clipped, so Huber equals least squares. The SmoothL1 button rescales the gradient to clip(r/β, ±1): the fit is identical, the loss reads 1/δ times smaller (1.628 → 0.543 at δ = 3).

Huber versus smooth L1 in PyTorch

PyTorch has both, and they differ only by a factor of the threshold. nn.HuberLoss(delta=δ) is ℓδ\ell_\delta above. nn.SmoothL1Loss(beta=β) is ℓβ/β\ell_\beta/\beta:

SmoothL1β(r)={0.5 r2β∣r∣<β∣r∣−0.5 β∣r∣≥β=ℓβ(r)βddrSmoothL1β=clip⁡ ⁣(rβ,−1,1)\text{SmoothL1}_\beta(r)=\begin{cases}\dfrac{0.5\,r^2}{\beta} & |r|<\beta\\[4pt] |r|-0.5\,\beta & |r|\ge\beta\end{cases}=\frac{\ell_\beta(r)}{\beta}\qquad\qquad \frac{d}{dr}\text{SmoothL1}_\beta=\operatorname{clip}\!\Big(\frac r\beta,-1,1\Big)
β\beta
same role as δ\delta: where the parabola ends. Default 1.0, so the two losses coincide at their defaults
slope 1\text{slope }1
smooth L1's linear part always has slope 1 (it is L1 with a rounded tip); Huber's has slope δ\delta
β=0\beta=0
allowed: smooth L1 becomes L1Loss. HuberLoss requires δ>0\delta>0

For a fixed threshold the two give the same minimizer, but the gradient magnitudes differ by δ\delta, so swapping one for the other silently changes the effective learning rate. The limits differ too. As β→∞\beta\to\infty smooth L1 goes to 0 (the parabola r2/2βr^2/2\beta flattens), while Huber goes to half the MSE. As the threshold goes to 0, smooth L1 becomes L1 while Huber goes to 0.

residual rℓ, δ = 1∂/∂ŷ, δ = 1ℓ, δ = 2∂/∂ŷ, δ = 2SmoothL1, β = 2∂/∂ŷ, β = 2
0.50.125−0.1250.125−0.1250.0625−0.0625
−10.5+0.250.5+0.250.25+0.125
21.5−0.252−0.51−0.25
87.5−0.2514−0.57−0.25
mean2.406254.156252.078125
Same four residuals as the MSE and MAE articles; mean over n = 4 (checked numerically in Node)

Reading the table

  1. 1δ=1\delta=1: only the first residual is inside the parabola, 12(0.5)2=0.125\tfrac12(0.5)^2=0.125. The others are linear, e.g. 1⋅(8−0.5)=7.51\cdot(8-0.5)=7.5. The mean 2.4062.406 matches the overview's Huberδ=1\text{Huber}_{\delta=1}, between MAE (2.8752.875) and half the MSE (8.6568.656).
  2. 2Gradients are clip⁡(y^−y,±δ)/n\operatorname{clip}(\hat y-y,\pm\delta)/n. With δ=1\delta=1 the outlier and the residual of 2 both get −0.25-0.25; under MSE the outlier's was −4-4.
  3. 3δ=2\delta=2: the residual of 2 sits exactly on the boundary, 12⋅22=2=2(2−1)\tfrac12\cdot2^2=2=2(2-1), and the outlier costs 2(8−1)=142(8-1)=14.
  4. 4Smooth L1 with β=2\beta=2 is the δ=2\delta=2 column halved, both value and gradient. At δ=β=1\delta=\beta=1 the two losses agree to the last digit.
python
1import torch, torch.nn as nn
2
3y = torch.tensor([3.0, -0.5, 2.0, 7.0])
4yhat = torch.tensor([2.5, 0.5, 0.0, -1.0], requires_grad=True)
5
6for d in (0.5, 1.0, 2.0):
7 h = nn.HuberLoss(delta=d)(yhat, y).item()
8 s = nn.SmoothL1Loss(beta=d)(yhat, y).item()
9 print(d, h, s, h / d)
10# 0.5 1.3125 2.625 2.625
11# 1.0 2.40625 2.40625 2.40625
12# 2.0 4.15625 2.078125 2.078125
13
14nn.HuberLoss(delta=2.0)(yhat, y).backward()
15print(yhat.grad) # tensor([-0.1250, 0.2500, -0.5000, -0.5000]) clip(ŷ − y, ±2) / 4
16yhat.grad = None
17nn.SmoothL1Loss(beta=2.0)(yhat, y).backward()
18print(yhat.grad) # tensor([-0.0625, 0.1250, -0.2500, -0.2500]) clip((ŷ − y)/2, ±1) / 4
19
20# Pick δ on the scale of the noise: e.g. from the residuals of a first fit
21res = (y - yhat).detach()
22sigma = 1.4826 * (res - res.median()).abs().median() # robust std from the MAD
23delta = 1.345 * sigma
HuberLoss vs SmoothL1Loss: same shape, scaled by the threshold

Use when

  • +Bounded influence: an outlier pulls with force at most δ
  • +Gradient fades near the optimum, so a fixed learning rate converges
  • +Convex and C¹; easy for every optimizer
  • +One knob interpolates between MSE (large δ) and MAE (small δ)

Avoid when

  • −δ is an extra hyperparameter, and it is scale-dependent
  • −Second derivative jumps at ±δ (use pseudo-Huber if that matters)
  • −Two PyTorch variants with different gradient scales are easy to confuse
  • −Still sensitive to outliers in the inputs (high leverage), not only the targets

Check yourself

Huber loss with δ = 1. One residual is 3; the network then gets worse on that example and the residual becomes 30. How does that example's gradient with respect to the prediction change?

Check yourself

A detector trained with nn.SmoothL1Loss(beta=0.1) is switched to nn.HuberLoss(delta=0.1) with everything else unchanged. What happens?

Remember

  • ✓ℓδ(r)=12r2\ell_\delta(r)=\tfrac12r^2 for ∣r∣≤δ|r|\le\delta, δ(∣r∣−12δ)\delta(|r|-\tfrac12\delta) beyond; gradient clip⁡(y^−y,±δ)\operatorname{clip}(\hat y-y,\pm\delta).
  • ✓δ→∞\delta\to\infty gives half the MSE, δ→0\delta\to0 (after dividing by δ\delta) gives MAE.
  • ✓Outliers beyond δ all pull with the same capped force, so moving them further does not move the fit; near the optimum it converges like MSE.
  • ✓PyTorch: HuberLoss(delta) =δ×=\delta\timesSmoothL1Loss(beta=delta); they coincide only at 1.
  • ✓Set δ\delta on the scale of the noise: standardized targets and δ=1\delta=1, or δ≈1.345σ^\delta\approx1.345\hat\sigma.
← back to Deep Learning roadmap