Definition
Mean absolute error (L1 loss) is the average of the absolute residuals, . Every unit of error costs the same, whether it belongs to a small miss or a huge one.
Where squared error charges an error of 10 a hundred times more than an error of 1, absolute error charges it ten times more: the price is linear, not quadratic. That one change has three consequences. The best constant prediction becomes the median instead of the mean, an outlier's pull on the model is capped, and the gradient no longer fades as the model gets close to the answer.
Absolute deviations are older than least squares. Boscovich (1757) fitted a line to measurements of the Earth's shape by minimizing the sum of absolute deviations, and Laplace (1774) proposed the two-sided exponential as a law of errors, the distribution that now bears his name. Least squares won the 19th century because it has a closed-form solution; absolute error returned with linear programming and robust statistics, and in deep learning, where everything is solved by gradient descent anyway, it costs no more than MSE.
Formula and subgradient
- residual; the per-element loss is
- if the prediction is too high, if too low: the size of the error is ignored
- the absolute value has a kink; its subdifferential is the interval . PyTorch uses
The absolute value is not differentiable at 0, but it is convex, and every convex function has subgradients: slopes of lines that touch the graph from below. At the kink any slope in qualifies. Gradient descent with a subgradient still converges for convex problems, but only if the step size shrinks, because the "gradient" does not shrink on its own. Since a real-valued residual is almost never exactly 0, the choice at the kink rarely matters; what matters is that is the same for an example that is off by 0.01 and one that is off by 100.
Properties
- Range
- , in the target's own units
- Minimizer
- the median; for a model, the conditional median of
- Noise model
- Laplace: is the NLL plus a constant
- Smoothness
- convex and continuous; not differentiable where a residual is 0; zero curvature elsewhere
- Influence of one point
- bounded: at most of gradient, however wrong it is
- Breakdown point
- 50% for the median (half the data can be corrupted before it moves arbitrarily) versus 0% for the mean
- PyTorch
nn.L1Loss(),F.l1_loss; alsonn.SmoothL1Loss(beta=0)
Why the median
Fit a single constant to targets . Away from the targets, is linear, and its slope just counts how many targets lie on each side:
- targets below the prediction: each pushes down with the same force
- the cumulative distribution of (given , for a model)
- : the optimum is the median
Each target votes by its side, not its distance. The slope is zero, or the subdifferential contains zero, exactly when half the targets are on each side. With an odd number of targets the minimum is the middle one; with an even number is flat between the two middle targets and any value there is optimal. Dragging the largest target from 10 to 1000 changes the mean by about 990/n but leaves the median, and the MAE-trained model, untouched.
Why absolute values: the Laplace likelihood
Model the noise as Laplace (double-exponential) with scale : . Then
- Laplace scale; the noise variance is
- constant in the network parameters when b is fixed
- the best scale is the MAE itself
With fixed, maximum likelihood is MAE minimization. The Laplace density has exponential tails, , much heavier than the Gaussian's : a residual of 6 scales is times as likely as a perfect fit instead of . Large residuals are not shocking under this model, so the fit does not bend toward them. Choosing MAE is asserting that your noise has heavy tails or occasional gross errors.
Worked example
| i | residual r | |r| | ∂L/∂ŷᵢ = sign(ŷᵢ − yᵢ)/4 | share of loss | MSE share |
|---|---|---|---|---|---|
| 1 | 0.5 | 0.5 | −0.25 | 4.3% | 0.4% |
| 2 | −1 | 1 | +0.25 | 8.7% | 1.4% |
| 3 | 2 | 2 | −0.25 | 17.4% | 5.8% |
| 4 | 8 | 8 | −0.25 | 69.6% | 92.4% |
- 1.
- 2All four gradients have magnitude . The outlier is pulled toward exactly as hard as the example that is off by 0.5; under MSE it was 16 times harder.
- 3Constant fit to targets : the mean 4 gives MAE ; every gives the minimum . Move the 10 to 1000 and the mean jumps to 251.5 while the MAE optimum stays .
1import math, torch, torch.nn as nn, torch.nn.functional as F23y = torch.tensor([3.0, -0.5, 2.0, 7.0])4yhat = torch.tensor([2.5, 0.5, 0.0, -1.0], requires_grad=True)5loss = nn.L1Loss()(yhat, y)6loss.backward()7print(loss.item()) # 2.8758print(yhat.grad) # tensor([-0.2500, 0.2500, -0.2500, -0.2500]) = sign(ŷ − y)/n910# Fit one constant to the figure's nine targets: it converges to the median, not the mean11ys = torch.tensor([0.6, 1.3, 1.9, 2.2, 2.6, 2.8, 3.1, 3.5, 4.4])12c = torch.tensor([7.5], requires_grad=True)13opt = torch.optim.SGD([c], lr=1.0)14sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda t: 1 / math.sqrt(t + 1)) # η/√t15for _ in range(40):16 opt.zero_grad()17 F.l1_loss(c.expand_as(ys), ys).backward()18 opt.step()19 sched.step() # drop this line and c bounces between 2.50 and 2.61 forever20print(c.item(), ys.median().item(), ys.mean().item()) # ≈2.60, 2.6, 2.492122# Exactly at the median the PyTorch subgradient is 0 (sign(0) = 0, four above, four below)Use when
- +Robust to outliers in the target: bounded influence, median breakdown point of 50%
- +In the target's own units; easy to interpret as an average miss
- +Matches heavy-tailed (Laplace) noise and estimates the conditional median
- +Same cost as MSE in any autodiff framework
Avoid when
- −Gradient magnitude does not shrink near the optimum: needs a decaying step size
- −Not differentiable at zero residual; zero curvature, so no second-order help
- −Less efficient than MSE when the noise really is Gaussian
- −Predicts the median, which is wrong if you need the mean (e.g. expected cost or revenue)
Check yourself
Targets are 1, 2, 3 and 10. Which constant predictions minimize the MAE?
Check yourself
You train with L1 loss and a constant learning rate. The loss drops quickly, then the validation curve keeps jittering at the same level. What is the most direct fix?
Remember
- ✓, subgradient ; PyTorch uses .
- ✓Minimizer is the median: each target votes by side, not distance, so outliers in y cannot drag it.
- ✓It is the negative log-likelihood of Laplace noise, whose heavier tails make large residuals unsurprising.
- ✓Constant gradient magnitude: fast far away, but it never settles without a decaying learning rate.
- ✓Generalizes to the pinball loss for any quantile; Huber blends it with MSE near zero.