# Huber loss

> Quadratic for residuals up to δ, linear beyond, so its gradient is the residual clipped to ±δ: MSE-like smooth convergence near the target, MAE-like bounded influence for outliers. PyTorch's SmoothL1Loss is the same curve divided by β.

Source: https://www.mlfoundry.in/learn/huber-loss  
Section: Deep Learning › Loss Functions  
Level: core · 10 min read · updated 2026-10-11

**Huber loss:** with threshold $\delta>0$ is $\tfrac12 r^2$ for residuals $|r|\le\delta$ and $\delta(|r|-\tfrac12\delta)$ beyond. The two pieces meet with the same value and the same slope at $|r|=\delta$, so the loss is smooth enough for gradient descent, behaves like MSE for small errors and like MAE for large ones.

MSE and MAE each have one good half. MSE's gradient fades near the optimum, so training settles, but it grows without bound for outliers. MAE's gradient is bounded, so outliers cannot take over, but it never fades, so training jitters. Huber keeps the good half of each: the parabola where errors are small and believable, the straight line where they are large and suspicious.

Peter Huber introduced it in "Robust Estimation of a Location Parameter" (Annals of Mathematical Statistics, 1964), the paper that founded robust statistics. He asked which loss gives the best worst-case estimate of a centre when a fraction $\varepsilon$ of Gaussian data may be replaced by arbitrary contamination, and the answer was this one, with $\delta$ set by $\varepsilon$. Deep learning rediscovered it twice: Girshick's Fast R-CNN (2015) used it under the name **smooth L1** for bounding-box regression because it is less sensitive to outliers than the L2 loss of earlier detectors, and DQN (Mnih et al., 2015) clipped the temporal-difference error to $[-1,1]$, which is exactly the gradient of Huber with $\delta=1$.

## Formula and gradient

$$
\ell_\delta(r)=\begin{cases}\tfrac12 r^2 & |r|\le\delta\\[2pt] \delta\big(|r|-\tfrac12\delta\big) & |r|>\delta\end{cases}\qquad\qquad \psi_\delta(r)=\frac{d\ell_\delta}{dr}=\operatorname{clip}(r,-\delta,\delta)=\begin{cases}r & |r|\le\delta\\ \delta\operatorname{sign}(r) & |r|>\delta\end{cases}
$$

- $r=y-\hat y$: residual; the gradient with respect to the prediction is $\partial\ell/\partial\hat y=-\psi_\delta(r)=\operatorname{clip}(\hat y-y,-\delta,\delta)$
- $\delta$: threshold, in the target's units: residuals up to $\delta$ are treated as noise, larger ones as outliers
- $\psi_\delta$: the influence function: how hard one example pulls on the fit. Bounded by $\delta$
- $-\tfrac12\delta^2$: the offset inside $\delta(|r|-\tfrac12\delta)$ that makes the two pieces meet: both equal $\tfrac12\delta^2$ at $|r|=\delta$

The function is continuous with a continuous first derivative, but its second derivative jumps from 1 to 0 at $|r|=\delta$. That is enough for first-order optimizers. Where a $C^\infty$ version is wanted, the **pseudo-Huber** loss $\delta^2\big(\sqrt{1+(r/\delta)^2}-1\big)$ has the same two regimes with a smooth transition.

The two limits recover the other losses. As $\delta\to\infty$ every residual is inside the parabola and $\ell_\delta=\tfrac12r^2$, half the squared error. As $\delta\to0$, $\ell_\delta/\delta\to|r|$: Huber divided by $\delta$ becomes MAE. That rescaled version is exactly PyTorch's smooth L1.

**Properties**

- **Range:** $[0,\infty)$; grows like $\tfrac12r^2$ near 0 and like $\delta|r|$ far away
- **Gradient:** $\operatorname{clip}(\hat y-y,-\delta,\delta)$: continuous, bounded by $\delta$
- **Smoothness:** convex, $C^1$; curvature 1 inside $\pm\delta$ and 0 outside
- **Minimizer:** an M-estimator between the mean ($\delta\to\infty$) and the median ($\delta\to0$)
- **Common δ:** 1.0 on standardized targets (PyTorch default); $1.345\hat\sigma$ in statistics, which keeps 95% efficiency on Gaussian data
- **PyTorch:** `nn.HuberLoss(delta=δ)` and `nn.SmoothL1Loss(beta=β)` $=\ell_\beta/\beta$

## Bounded influence: outliers count as if they were only δ away

Gradient descent moves the model by the sum of the per-example gradients. Under MSE an example 8 units off pulls 8 times harder than one 1 unit off. Under Huber both pull with force at most $\delta$: an outlier is still allowed to say "the fit is too low here", but not how much too low. Its pull stops growing once it is farther than $\delta$, which is why sliding an outlier further away has no effect on a Huber fit.

The same fact gives a classic solver. Write $\psi_\delta(r)=w(r)\,r$ with weight $w(r)=\min(1,\delta/|r|)$. Setting the gradient to zero is then a **weighted** least-squares problem, so Huber regression can be solved by iteratively reweighted least squares (IRLS): fit, recompute the weights from the residuals, refit. Points inside $\pm\delta$ get weight 1; a point at distance $5\delta$ gets weight 0.2.

> Figure: Fourteen points around the line y = 1 + 0.5x (noise σ = 0.3), two of them pulled down by 6. Least squares (blue) is dragged to slope 0.160; Huber with δ = 1 (green, solved by IRLS) keeps 0.404, and the two outliers get weights 0.19 and 0.18 (dot opacity). Pull them down to 10: the least-squares slope falls to −0.058 while the Huber line does not move at all, because both outliers sit on the flat part of the clipped gradient (right). Shrink δ to 0.4 (about 1.345σ) and the slope recovers to 0.452; raise δ past 4 and no residual is clipped, so Huber equals least squares. The SmoothL1 button rescales the gradient to clip(r/β, ±1): the fit is identical, the loss reads 1/δ times smaller (1.628 → 0.543 at δ = 3).

## Huber versus smooth L1 in PyTorch

PyTorch has both, and they differ only by a factor of the threshold. `nn.HuberLoss(delta=δ)` is $\ell_\delta$ above. `nn.SmoothL1Loss(beta=β)` is $\ell_\beta/\beta$:

$$
\text{SmoothL1}_\beta(r)=\begin{cases}\dfrac{0.5\,r^2}{\beta} & |r|<\beta\\[4pt] |r|-0.5\,\beta & |r|\ge\beta\end{cases}=\frac{\ell_\beta(r)}{\beta}\qquad\qquad \frac{d}{dr}\text{SmoothL1}_\beta=\operatorname{clip}\!\Big(\frac r\beta,-1,1\Big)
$$

- $\beta$: same role as $\delta$: where the parabola ends. Default 1.0, so the two losses coincide at their defaults
- $\text{slope }1$: smooth L1's linear part always has slope 1 (it is L1 with a rounded tip); Huber's has slope $\delta$
- $\beta=0$: allowed: smooth L1 becomes `L1Loss`. `HuberLoss` requires $\delta>0$

For a fixed threshold the two give the same minimizer, but the gradient magnitudes differ by $\delta$, so swapping one for the other silently changes the effective learning rate. The limits differ too. As $\beta\to\infty$ smooth L1 goes to 0 (the parabola $r^2/2\beta$ flattens), while Huber goes to half the MSE. As the threshold goes to 0, smooth L1 becomes L1 while Huber goes to 0.

Same four residuals as the MSE and MAE articles; mean over n = 4 (checked numerically in Node)

| residual r | ℓ, δ = 1 | ∂/∂ŷ, δ = 1 | ℓ, δ = 2 | ∂/∂ŷ, δ = 2 | SmoothL1, β = 2 | ∂/∂ŷ, β = 2 |
| --- | --- | --- | --- | --- | --- | --- |
| 0.5 | 0.125 | −0.125 | 0.125 | −0.125 | 0.0625 | −0.0625 |
| −1 | 0.5 | +0.25 | 0.5 | +0.25 | 0.25 | +0.125 |
| 2 | 1.5 | −0.25 | 2 | −0.5 | 1 | −0.25 |
| 8 | 7.5 | −0.25 | 14 | −0.5 | 7 | −0.25 |
| **mean** | **2.40625** |  | **4.15625** |  | **2.078125** |  |

**Reading the table**

1. $\delta=1$: only the first residual is inside the parabola, $\tfrac12(0.5)^2=0.125$. The others are linear, e.g. $1\cdot(8-0.5)=7.5$. The mean $2.406$ matches the overview's $\text{Huber}_{\delta=1}$, between MAE ($2.875$) and half the MSE ($8.656$).
2. Gradients are $\operatorname{clip}(\hat y-y,\pm\delta)/n$. With $\delta=1$ the outlier and the residual of 2 both get $-0.25$; under MSE the outlier's was $-4$.
3. $\delta=2$: the residual of 2 sits exactly on the boundary, $\tfrac12\cdot2^2=2=2(2-1)$, and the outlier costs $2(8-1)=14$.
4. Smooth L1 with $\beta=2$ is the $\delta=2$ column halved, both value and gradient. At $\delta=\beta=1$ the two losses agree to the last digit.

```python
import torch, torch.nn as nn

y    = torch.tensor([3.0, -0.5, 2.0,  7.0])
yhat = torch.tensor([2.5,  0.5, 0.0, -1.0], requires_grad=True)

for d in (0.5, 1.0, 2.0):
    h = nn.HuberLoss(delta=d)(yhat, y).item()
    s = nn.SmoothL1Loss(beta=d)(yhat, y).item()
    print(d, h, s, h / d)
# 0.5  1.3125   2.625     2.625
# 1.0  2.40625  2.40625   2.40625
# 2.0  4.15625  2.078125  2.078125

nn.HuberLoss(delta=2.0)(yhat, y).backward()
print(yhat.grad)     # tensor([-0.1250,  0.2500, -0.5000, -0.5000])   clip(ŷ − y, ±2) / 4
yhat.grad = None
nn.SmoothL1Loss(beta=2.0)(yhat, y).backward()
print(yhat.grad)     # tensor([-0.0625,  0.1250, -0.2500, -0.2500])   clip((ŷ − y)/2, ±1) / 4

# Pick δ on the scale of the noise: e.g. from the residuals of a first fit
res = (y - yhat).detach()
sigma = 1.4826 * (res - res.median()).abs().median()   # robust std from the MAD
delta = 1.345 * sigma
```

HuberLoss vs SmoothL1Loss: same shape, scaled by the threshold

> **δ has units:** $\delta$ is measured in the target's units, so the same value means very different things for targets in metres and in millimetres. Standardize the targets first, then $\delta=1$ means "one standard deviation"; or estimate the noise scale robustly ($\hat\sigma=1.4826\cdot\mathrm{MAD}$) and set $\delta\approx1.345\hat\sigma$. And when you swap `HuberLoss` for `SmoothL1Loss` with $\delta\neq1$, retune the learning rate: the gradients differ by a factor $\delta$.

> **Huber is MAE with the jitter removed:** Near the optimum all residuals are small, so Huber is a parabola and its gradient shrinks like MSE's: a constant learning rate converges instead of oscillating. Far from the optimum, or for genuine outliers, its gradient is capped like MAE's. Gradient clipping applied per example to the loss gradient, as in DQN, is the same idea, which is why "clip the TD error" and "use Huber loss" are interchangeable.

> **Where you meet it:** Bounding-box regression in Fast/Faster R-CNN and many detectors (`SmoothL1Loss`, often with $\beta<1$); value regression in DQN and its descendants (the PyTorch DQN tutorial uses `nn.SmoothL1Loss`); robust linear models (scikit-learn's `HuberRegressor`, default $\varepsilon=1.35$); depth and flow estimation, where a few pixels have wildly wrong labels.

**Strengths**

- Bounded influence: an outlier pulls with force at most δ
- Gradient fades near the optimum, so a fixed learning rate converges
- Convex and C¹; easy for every optimizer
- One knob interpolates between MSE (large δ) and MAE (small δ)

**Limitations**

- δ is an extra hyperparameter, and it is scale-dependent
- Second derivative jumps at ±δ (use pseudo-Huber if that matters)
- Two PyTorch variants with different gradient scales are easy to confuse
- Still sensitive to outliers in the inputs (high leverage), not only the targets

**Question:** Huber loss with δ = 1. One residual is 3; the network then gets worse on that example and the residual becomes 30. How does that example's gradient with respect to the prediction change?

- A. It grows 10 times
- B. It stays the same: it was already clipped to magnitude 1
- C. It drops to zero
- D. It grows 100 times

**Answer:** B. It stays the same: it was already clipped to magnitude 1 — $\partial\ell/\partial\hat y=\operatorname{clip}(\hat y-y,-1,1)$. Both $|r|=3$ and $|r|=30$ are beyond $\delta=1$, so the gradient is $-1$ in both cases. Under MSE it would grow from 6 to 60 in magnitude.

**Question:** A detector trained with nn.SmoothL1Loss(beta=0.1) is switched to nn.HuberLoss(delta=0.1) with everything else unchanged. What happens?

- A. Nothing: they are the same loss
- B. The minimizer changes completely
- C. Gradients become 10 times smaller, as if the learning rate were divided by 10
- D. Gradients become 10 times larger

**Answer:** C. Gradients become 10 times smaller, as if the learning rate were divided by 10 — $\text{SmoothL1}_\beta=\ell_\beta/\beta$, so $\text{Huber}_{0.1}=0.1\times\text{SmoothL1}_{0.1}$ and every gradient shrinks by the same factor 0.1. The optimum is the same; with plain SGD the effective step is 10 times smaller (adaptive optimizers like Adam largely cancel the rescaling).

**Key points**

- $\ell_\delta(r)=\tfrac12r^2$ for $|r|\le\delta$, $\delta(|r|-\tfrac12\delta)$ beyond; gradient $\operatorname{clip}(\hat y-y,\pm\delta)$.
- $\delta\to\infty$ gives half the MSE, $\delta\to0$ (after dividing by $\delta$) gives MAE.
- Outliers beyond δ all pull with the same capped force, so moving them further does not move the fit; near the optimum it converges like MSE.
- PyTorch: `HuberLoss(delta)` $=\delta\times$`SmoothL1Loss(beta=delta)`; they coincide only at 1.
- Set $\delta$ on the scale of the noise: standardized targets and $\delta=1$, or $\delta\approx1.345\hat\sigma$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
