Mean absolute error (MAE)

The average absolute residual. It is the Laplace negative log-likelihood and its minimizer is the median, so outliers barely move it, but its gradient is a constant ±1 that never shrinks, so training needs a decaying step size.

9 min read814 wordscore0 viewsupdated 2026-10-11

Definition

Mean absolute error (L1 loss) is the average of the absolute residuals, MAE=1n∑i∣yi−y^i∣\mathrm{MAE}=\frac1n\sum_i |y_i-\hat y_i|. Every unit of error costs the same, whether it belongs to a small miss or a huge one.

Where squared error charges an error of 10 a hundred times more than an error of 1, absolute error charges it ten times more: the price is linear, not quadratic. That one change has three consequences. The best constant prediction becomes the median instead of the mean, an outlier's pull on the model is capped, and the gradient no longer fades as the model gets close to the answer.

Absolute deviations are older than least squares. Boscovich (1757) fitted a line to measurements of the Earth's shape by minimizing the sum of absolute deviations, and Laplace (1774) proposed the two-sided exponential as a law of errors, the distribution that now bears his name. Least squares won the 19th century because it has a closed-form solution; absolute error returned with linear programming and robust statistics, and in deep learning, where everything is solved by gradient descent anyway, it costs no more than MSE.

Formula and subgradient

LMAE=1n∑i=1n∣yi−y^i∣∂LMAE∂y^i=1nsign⁡(y^i−yi)∈{−1n, 0, 1n}\mathcal L_{\text{MAE}}=\frac1n\sum_{i=1}^{n}|y_i-\hat y_i|\qquad\qquad \frac{\partial\mathcal L_{\text{MAE}}}{\partial\hat y_i}=\frac1n\operatorname{sign}(\hat y_i-y_i)\in\Big\{-\tfrac1n,\,0,\,\tfrac1n\Big\}
ri=yi−y^ir_i=y_i-\hat y_i
residual; the per-element loss is ∣ri∣|r_i|
sign⁡(y^i−yi)\operatorname{sign}(\hat y_i-y_i)
+1+1 if the prediction is too high, −1-1 if too low: the size of the error is ignored
∂∣r∣ at 0\partial|r|\text{ at }0
the absolute value has a kink; its subdifferential is the interval [−1,1][-1,1]. PyTorch uses sign⁡(0)=0\operatorname{sign}(0)=0

The absolute value is not differentiable at 0, but it is convex, and every convex function has subgradients: slopes of lines that touch the graph from below. At the kink any slope in [−1,1][-1,1] qualifies. Gradient descent with a subgradient still converges for convex problems, but only if the step size shrinks, because the "gradient" does not shrink on its own. Since a real-valued residual is almost never exactly 0, the choice at the kink rarely matters; what matters is that ∣∂L/∂y^i∣|\partial\mathcal L/\partial\hat y_i| is the same 1/n1/n for an example that is off by 0.01 and one that is off by 100.

Properties

Range
[0,∞)[0,\infty), in the target's own units
Minimizer
the median; for a model, the conditional median of y∣xy\mid x
Noise model
Laplace: MAE\mathrm{MAE} is b×b\times the NLL plus a constant
Smoothness
convex and continuous; not differentiable where a residual is 0; zero curvature elsewhere
Influence of one point
bounded: at most 1/n1/n of gradient, however wrong it is
Breakdown point
50% for the median (half the data can be corrupted before it moves arbitrarily) versus 0% for the mean
PyTorch
nn.L1Loss(), F.l1_loss; also nn.SmoothL1Loss(beta=0)

Why the median

Fit a single constant cc to targets y1,…,yny_1,\dots,y_n. Away from the targets, L(c)=1n∑i∣yi−c∣L(c)=\frac1n\sum_i|y_i-c| is linear, and its slope just counts how many targets lie on each side:

dLdc=#{yi<c}−#{yi>c}n⟹ddc E ∣Y−c∣=P(Y<c)−P(Y>c)=2F(c)−1\frac{dL}{dc}=\frac{\#\{y_i<c\}-\#\{y_i>c\}}{n}\qquad\Longrightarrow\qquad \frac{d}{dc}\,\mathbb E\,|Y-c|=P(Y<c)-P(Y>c)=2F(c)-1
#{yi<c}\#\{y_i<c\}
targets below the prediction: each pushes cc down with the same force 1/n1/n
FF
the cumulative distribution of YY (given xx, for a model)
2F(c)−1=02F(c)-1=0
F(c)=12F(c)=\tfrac12: the optimum is the median

Each target votes ±1/n\pm1/n by its side, not its distance. The slope is zero, or the subdifferential contains zero, exactly when half the targets are on each side. With an odd number of targets the minimum is the middle one; with an even number LL is flat between the two middle targets and any value there is optimal. Dragging the largest target from 10 to 1000 changes the mean by about 990/n but leaves the median, and the MAE-trained model, untouched.

targets yᵢ, prediction c, residuals; bump = Laplace(c, b=1), −log of it = loss02468101214L(c) = mean |yᵢ − c|: a kink at every target, minimum at the median02468median 2.60mean 2.49∂L/∂c = (#below − #above)/n: steps of 2/n = 0.222, zero only at the median−101c = 7.500 · MAE = 5.011 · ∂L/∂c = 1.000press run to take subgradient steps c ← c − η·∂L/∂c
fig.Nine targets and one constant prediction c. L(c) is a polyline with a kink at each target, and its slope (bottom) climbs in steps of 2/9 = 0.222 from −1 to +1, crossing zero at the median 2.60. Drag the red outlier y₉ from 2.8 to 12: the mean moves from 2.49 to 3.51, the median stays at 2.60. Press run: from c = 7.5 with constant step η = 1 the slope is +1 until c passes the targets, then the iterates bounce between 2.500 and 2.611 forever, because the subgradient next to the median is still ±1/9. Switch on decay η/√t and the bounce shrinks (the last two of 40 iterates are 2.614 and 2.597).

Why absolute values: the Laplace likelihood

Model the noise as Laplace (double-exponential) with scale bb: p(y∣x)=12bexp⁡ ⁣(−∣y−y^∣/b)p(y\mid x)=\frac{1}{2b}\exp\!\big(-|y-\hat y|/b\big). Then

−ln⁡p(y∣x)=∣y−y^∣b+ln⁡(2b)b^MLE=1n∑i∣yi−y^i∣-\ln p(y\mid x)=\frac{|y-\hat y|}{b}+\ln(2b)\qquad\qquad \hat b_{\text{MLE}}=\frac1n\sum_i|y_i-\hat y_i|
bb
Laplace scale; the noise variance is 2b22b^2
ln⁡(2b)\ln(2b)
constant in the network parameters when b is fixed
b^MLE\hat b_{\text{MLE}}
the best scale is the MAE itself

With bb fixed, maximum likelihood is MAE minimization. The Laplace density has exponential tails, e−∣r∣e^{-|r|}, much heavier than the Gaussian's e−r2/2e^{-r^2/2}: a residual of 6 scales is e−6≈0.0025e^{-6}\approx0.0025 times as likely as a perfect fit instead of 1.5×10−81.5\times10^{-8}. Large residuals are not shocking under this model, so the fit does not bend toward them. Choosing MAE is asserting that your noise has heavy tails or occasional gross errors.

Worked example

iresidual r|r|∂L/∂ŷᵢ = sign(ŷᵢ − yᵢ)/4share of lossMSE share
10.50.5−0.254.3%0.4%
2−11+0.258.7%1.4%
322−0.2517.4%5.8%
488−0.2569.6%92.4%
Same data as the MSE article: y=(3, −0.5, 2, 7)y=(3,\,-0.5,\,2,\,7), y^=(2.5, 0.5, 0, −1)\hat y=(2.5,\,0.5,\,0,\,-1), n=4n=4 (checked numerically in Node)
  1. 1MAE=(0.5+1+2+8)/4=2.875\mathrm{MAE}=(0.5+1+2+8)/4=2.875.
  2. 2All four gradients have magnitude 0.250.25. The outlier is pulled toward exactly as hard as the example that is off by 0.5; under MSE it was 16 times harder.
  3. 3Constant fit to targets 1,2,3,101,2,3,10: the mean 4 gives MAE =(3+2+1+6)/4=3=(3+2+1+6)/4=3; every c∈[2,3]c\in[2,3] gives the minimum 2.52.5. Move the 10 to 1000 and the mean jumps to 251.5 while the MAE optimum stays [2,3][2,3].
python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3y = torch.tensor([3.0, -0.5, 2.0, 7.0])
4yhat = torch.tensor([2.5, 0.5, 0.0, -1.0], requires_grad=True)
5loss = nn.L1Loss()(yhat, y)
6loss.backward()
7print(loss.item()) # 2.875
8print(yhat.grad) # tensor([-0.2500, 0.2500, -0.2500, -0.2500]) = sign(ŷ − y)/n
9
10# Fit one constant to the figure's nine targets: it converges to the median, not the mean
11ys = torch.tensor([0.6, 1.3, 1.9, 2.2, 2.6, 2.8, 3.1, 3.5, 4.4])
12c = torch.tensor([7.5], requires_grad=True)
13opt = torch.optim.SGD([c], lr=1.0)
14sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda t: 1 / math.sqrt(t + 1)) # η/√t
15for _ in range(40):
16 opt.zero_grad()
17 F.l1_loss(c.expand_as(ys), ys).backward()
18 opt.step()
19 sched.step() # drop this line and c bounces between 2.50 and 2.61 forever
20print(c.item(), ys.median().item(), ys.mean().item()) # ≈2.60, 2.6, 2.49
21
22# Exactly at the median the PyTorch subgradient is 0 (sign(0) = 0, four above, four below)
L1 loss, its sign gradient, and subgradient descent to the median

Use when

  • +Robust to outliers in the target: bounded influence, median breakdown point of 50%
  • +In the target's own units; easy to interpret as an average miss
  • +Matches heavy-tailed (Laplace) noise and estimates the conditional median
  • +Same cost as MSE in any autodiff framework

Avoid when

  • −Gradient magnitude does not shrink near the optimum: needs a decaying step size
  • −Not differentiable at zero residual; zero curvature, so no second-order help
  • −Less efficient than MSE when the noise really is Gaussian
  • −Predicts the median, which is wrong if you need the mean (e.g. expected cost or revenue)

Check yourself

Targets are 1, 2, 3 and 10. Which constant predictions minimize the MAE?

Check yourself

You train with L1 loss and a constant learning rate. The loss drops quickly, then the validation curve keeps jittering at the same level. What is the most direct fix?

Remember

  • ✓MAE=1n∑∣y−y^∣\mathrm{MAE}=\frac1n\sum|y-\hat y|, subgradient 1nsign⁡(y^−y)\frac1n\operatorname{sign}(\hat y-y); PyTorch uses sign⁡(0)=0\operatorname{sign}(0)=0.
  • ✓Minimizer is the median: each target votes by side, not distance, so outliers in y cannot drag it.
  • ✓It is the negative log-likelihood of Laplace noise, whose heavier tails make large residuals unsurprising.
  • ✓Constant gradient magnitude: fast far away, but it never settles without a decaying learning rate.
  • ✓Generalizes to the pinball loss for any quantile; Huber blends it with MSE near zero.
← back to Deep Learning roadmap