# Mean absolute error (MAE)

> The average absolute residual. It is the Laplace negative log-likelihood and its minimizer is the median, so outliers barely move it, but its gradient is a constant ±1 that never shrinks, so training needs a decaying step size.

Source: https://www.mlfoundry.in/learn/mean-absolute-error-mae  
Section: Deep Learning › Loss Functions  
Level: core · 9 min read · updated 2026-10-11

**Mean absolute error:** (L1 loss) is the average of the absolute residuals, $\mathrm{MAE}=\frac1n\sum_i |y_i-\hat y_i|$. Every unit of error costs the same, whether it belongs to a small miss or a huge one.

Where squared error charges an error of 10 a hundred times more than an error of 1, absolute error charges it ten times more: the price is linear, not quadratic. That one change has three consequences. The best constant prediction becomes the **median** instead of the mean, an outlier's pull on the model is capped, and the gradient no longer fades as the model gets close to the answer.

Absolute deviations are older than least squares. Boscovich (1757) fitted a line to measurements of the Earth's shape by minimizing the sum of absolute deviations, and Laplace (1774) proposed the two-sided exponential as a law of errors, the distribution that now bears his name. Least squares won the 19th century because it has a closed-form solution; absolute error returned with linear programming and robust statistics, and in deep learning, where everything is solved by gradient descent anyway, it costs no more than MSE.

## Formula and subgradient

$$
\mathcal L_{\text{MAE}}=\frac1n\sum_{i=1}^{n}|y_i-\hat y_i|\qquad\qquad \frac{\partial\mathcal L_{\text{MAE}}}{\partial\hat y_i}=\frac1n\operatorname{sign}(\hat y_i-y_i)\in\Big\{-\tfrac1n,\,0,\,\tfrac1n\Big\}
$$

- $r_i=y_i-\hat y_i$: residual; the per-element loss is $|r_i|$
- $\operatorname{sign}(\hat y_i-y_i)$: $+1$ if the prediction is too high, $-1$ if too low: the size of the error is ignored
- $\partial|r|\text{ at }0$: the absolute value has a kink; its subdifferential is the interval $[-1,1]$. PyTorch uses $\operatorname{sign}(0)=0$

The absolute value is not differentiable at 0, but it is convex, and every convex function has **subgradients**: slopes of lines that touch the graph from below. At the kink any slope in $[-1,1]$ qualifies. Gradient descent with a subgradient still converges for convex problems, but only if the step size shrinks, because the "gradient" does not shrink on its own. Since a real-valued residual is almost never exactly 0, the choice at the kink rarely matters; what matters is that $|\partial\mathcal L/\partial\hat y_i|$ is the same $1/n$ for an example that is off by 0.01 and one that is off by 100.

**Properties**

- **Range:** $[0,\infty)$, in the target's own units
- **Minimizer:** the median; for a model, the conditional median of $y\mid x$
- **Noise model:** Laplace: $\mathrm{MAE}$ is $b\times$ the NLL plus a constant
- **Smoothness:** convex and continuous; not differentiable where a residual is 0; zero curvature elsewhere
- **Influence of one point:** bounded: at most $1/n$ of gradient, however wrong it is
- **Breakdown point:** 50% for the median (half the data can be corrupted before it moves arbitrarily) versus 0% for the mean
- **PyTorch:** `nn.L1Loss()`, `F.l1_loss`; also `nn.SmoothL1Loss(beta=0)`

## Why the median

Fit a single constant $c$ to targets $y_1,\dots,y_n$. Away from the targets, $L(c)=\frac1n\sum_i|y_i-c|$ is linear, and its slope just counts how many targets lie on each side:

$$
\frac{dL}{dc}=\frac{\#\{y_i<c\}-\#\{y_i>c\}}{n}\qquad\Longrightarrow\qquad \frac{d}{dc}\,\mathbb E\,|Y-c|=P(Y<c)-P(Y>c)=2F(c)-1
$$

- $\#\{y_i<c\}$: targets below the prediction: each pushes $c$ down with the same force $1/n$
- $F$: the cumulative distribution of $Y$ (given $x$, for a model)
- $2F(c)-1=0$: $F(c)=\tfrac12$: the optimum is the median

Each target votes $\pm1/n$ by its **side**, not its distance. The slope is zero, or the subdifferential contains zero, exactly when half the targets are on each side. With an odd number of targets the minimum is the middle one; with an even number $L$ is flat between the two middle targets and any value there is optimal. Dragging the largest target from 10 to 1000 changes the mean by about 990/n but leaves the median, and the MAE-trained model, untouched.

> Figure: Nine targets and one constant prediction c. L(c) is a polyline with a kink at each target, and its slope (bottom) climbs in steps of 2/9 = 0.222 from −1 to +1, crossing zero at the median 2.60. Drag the red outlier y₉ from 2.8 to 12: the mean moves from 2.49 to 3.51, the median stays at 2.60. Press run: from c = 7.5 with constant step η = 1 the slope is +1 until c passes the targets, then the iterates bounce between 2.500 and 2.611 forever, because the subgradient next to the median is still ±1/9. Switch on decay η/√t and the bounce shrinks (the last two of 40 iterates are 2.614 and 2.597).

## Why absolute values: the Laplace likelihood

Model the noise as Laplace (double-exponential) with scale $b$: $p(y\mid x)=\frac{1}{2b}\exp\!\big(-|y-\hat y|/b\big)$. Then

$$
-\ln p(y\mid x)=\frac{|y-\hat y|}{b}+\ln(2b)\qquad\qquad \hat b_{\text{MLE}}=\frac1n\sum_i|y_i-\hat y_i|
$$

- $b$: Laplace scale; the noise variance is $2b^2$
- $\ln(2b)$: constant in the network parameters when b is fixed
- $\hat b_{\text{MLE}}$: the best scale is the MAE itself

With $b$ fixed, maximum likelihood is MAE minimization. The Laplace density has exponential tails, $e^{-|r|}$, much heavier than the Gaussian's $e^{-r^2/2}$: a residual of 6 scales is $e^{-6}\approx0.0025$ times as likely as a perfect fit instead of $1.5\times10^{-8}$. Large residuals are not shocking under this model, so the fit does not bend toward them. Choosing MAE is asserting that your noise has heavy tails or occasional gross errors.

## Worked example

Same data as the MSE article: $y=(3,\,-0.5,\,2,\,7)$, $\hat y=(2.5,\,0.5,\,0,\,-1)$, $n=4$ (checked numerically in Node)

| i | residual r | \|r\| | ∂L/∂ŷᵢ = sign(ŷᵢ − yᵢ)/4 | share of loss | MSE share |
| --- | --- | --- | --- | --- | --- |
| 1 | 0.5 | 0.5 | −0.25 | 4.3% | 0.4% |
| 2 | −1 | 1 | +0.25 | 8.7% | 1.4% |
| 3 | 2 | 2 | −0.25 | 17.4% | 5.8% |
| 4 | 8 | 8 | −0.25 | 69.6% | 92.4% |

1. $\mathrm{MAE}=(0.5+1+2+8)/4=2.875$.
2. All four gradients have magnitude $0.25$. The outlier is pulled toward exactly as hard as the example that is off by 0.5; under MSE it was 16 times harder.
3. Constant fit to targets $1,2,3,10$: the mean 4 gives MAE $=(3+2+1+6)/4=3$; every $c\in[2,3]$ gives the minimum $2.5$. Move the 10 to 1000 and the mean jumps to 251.5 while the MAE optimum stays $[2,3]$.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

y    = torch.tensor([3.0, -0.5, 2.0,  7.0])
yhat = torch.tensor([2.5,  0.5, 0.0, -1.0], requires_grad=True)
loss = nn.L1Loss()(yhat, y)
loss.backward()
print(loss.item())        # 2.875
print(yhat.grad)          # tensor([-0.2500,  0.2500, -0.2500, -0.2500])  = sign(ŷ − y)/n

# Fit one constant to the figure's nine targets: it converges to the median, not the mean
ys = torch.tensor([0.6, 1.3, 1.9, 2.2, 2.6, 2.8, 3.1, 3.5, 4.4])
c = torch.tensor([7.5], requires_grad=True)
opt = torch.optim.SGD([c], lr=1.0)
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda t: 1 / math.sqrt(t + 1))  # η/√t
for _ in range(40):
    opt.zero_grad()
    F.l1_loss(c.expand_as(ys), ys).backward()
    opt.step()
    sched.step()          # drop this line and c bounces between 2.50 and 2.61 forever
print(c.item(), ys.median().item(), ys.mean().item())   # ≈2.60, 2.6, 2.49

# Exactly at the median the PyTorch subgradient is 0 (sign(0) = 0, four above, four below)
```

L1 loss, its sign gradient, and subgradient descent to the median

> **It never settles by itself:** MSE's gradient fades as the error shrinks, so a fixed learning rate gives smooth convergence. MAE's does not: near the optimum the update is still $\eta/n$ per example, so plain SGD with a constant step oscillates around the median with an amplitude set by $\eta$. Use a decaying schedule (step, cosine, $1/\sqrt t$) or an adaptive optimizer, or switch to Huber, which is MAE far away and MSE close in.

> **Robust, not magical:** MAE bounds the influence of an outlier in $y$: its gradient can never exceed $1/n$. It does **not** protect against an outlier in $x$ (a high-leverage input), because the loss still depends on $\hat y(x)$ and a far-away input can swing a linear model's slope. It also discards information when the noise really is Gaussian: the median is a less efficient estimator than the mean there (asymptotic efficiency $2/\pi\approx64\%$).

> **Quantiles, pix2pix and forecasting:** MAE is the $\tau=0.5$ case of the **pinball loss** $\rho_\tau(r)=\max(\tau r,(\tau-1)r)$ of quantile regression (Koenker and Bassett, 1978), which estimates any conditional quantile; $\rho_{0.5}=\tfrac12|r|$. Image-to-image models such as pix2pix (Isola et al., 2017) use an L1 reconstruction term because, as the authors report, it blurs less than L2. In forecasting, MAE is a standard metric because it is in the data's units and easy to explain.

**Strengths**

- Robust to outliers in the target: bounded influence, median breakdown point of 50%
- In the target's own units; easy to interpret as an average miss
- Matches heavy-tailed (Laplace) noise and estimates the conditional median
- Same cost as MSE in any autodiff framework

**Limitations**

- Gradient magnitude does not shrink near the optimum: needs a decaying step size
- Not differentiable at zero residual; zero curvature, so no second-order help
- Less efficient than MSE when the noise really is Gaussian
- Predicts the median, which is wrong if you need the mean (e.g. expected cost or revenue)

**Question:** Targets are 1, 2, 3 and 10. Which constant predictions minimize the MAE?

- A. Only c = 2.5
- B. Every c between 2 and 3
- C. Only c = 4 (the mean)
- D. Only c = 2

**Answer:** B. Every c between 2 and 3 — For $2<c<3$ two targets lie on each side, so the slope $(\#\text{below}-\#\text{above})/n$ is 0 and $L$ is flat at its minimum $2.5$. With an even number of targets the median is a whole interval; conventions such as the midpoint 2.5 or `torch.median`'s lower value 2 just pick one point of it.

**Question:** You train with L1 loss and a constant learning rate. The loss drops quickly, then the validation curve keeps jittering at the same level. What is the most direct fix?

- A. Increase the learning rate
- B. Decay the learning rate (or switch to Huber)
- C. Add more layers
- D. Switch to MSE to make the model more robust

**Answer:** B. Decay the learning rate (or switch to Huber) — The subgradient of $|r|$ has magnitude 1 however small the residual, so with a fixed step the parameters keep overshooting the median by an amount proportional to the step. A decaying step makes the overshoot vanish; Huber replaces the kink with a parabola so the gradient fades near 0.

**Key points**

- $\mathrm{MAE}=\frac1n\sum|y-\hat y|$, subgradient $\frac1n\operatorname{sign}(\hat y-y)$; PyTorch uses $\operatorname{sign}(0)=0$.
- Minimizer is the median: each target votes by side, not distance, so outliers in y cannot drag it.
- It is the negative log-likelihood of Laplace noise, whose heavier tails make large residuals unsurprising.
- Constant gradient magnitude: fast far away, but it never settles without a decaying learning rate.
- Generalizes to the pinball loss for any quantile; Huber blends it with MSE near zero.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
