Mean squared error (MSE)

The average squared residual. It is the Gaussian negative log-likelihood, its minimizer is the conditional mean, and its gradient grows linearly with the error, so a single outlier can dominate training.

10 min read876 wordscore0 viewsupdated 2026-10-11

Definition

Mean squared error is the average of the squared residuals, MSE=1n∑i(yi−y^i)2\mathrm{MSE}=\frac1n\sum_i (y_i-\hat y_i)^2. It is the default loss for regression: smooth, convex in the prediction, and the loss you get from maximum likelihood when the noise on the targets is Gaussian.

Squaring does two jobs. It makes every error positive, and it makes big errors count much more than small ones: an error of 2 costs 4 times as much as an error of 1, and an error of 10 costs 100 times as much. That is exactly right when errors really are Gaussian, and exactly wrong when the data contain outliers.

The idea is over two centuries old. Legendre published the method of least squares in 1805 for fitting comet orbits. Gauss (1809) justified it probabilistically: least squares gives the most probable parameters when the errors follow what we now call the normal distribution, and he claimed to have used it since 1795. Every linear regression, every autoencoder trained on pixel error and the simplified training objective of DDPM diffusion models (Ho et al., 2020), which regresses the added noise, minimize this same quantity.

Formula and gradient

LMSE=1n∑i=1n(yi−y^i)2∂LMSE∂y^i=2n(y^i−yi)∂2LMSE∂y^i2=2n\mathcal L_{\text{MSE}}=\frac1n\sum_{i=1}^{n}(y_i-\hat y_i)^2\qquad\qquad \frac{\partial\mathcal L_{\text{MSE}}}{\partial\hat y_i}=\frac{2}{n}(\hat y_i-y_i)\qquad\qquad \frac{\partial^2\mathcal L_{\text{MSE}}}{\partial\hat y_i^2}=\frac2n
yi, y^iy_i,\ \hat y_i
target and network output for element ii
ri=yi−y^ir_i=y_i-\hat y_i
residual; the per-element loss is ri2r_i^2
nn
number of elements averaged over (with reduction="mean" PyTorch divides by all elements, batch × outputs)
2n(y^i−yi)\tfrac2n(\hat y_i-y_i)
the error signal sent into backprop: proportional to the residual, so it is unbounded

The gradient is linear in the error. Far from the target it is large, so learning is fast; near the target it shrinks to zero, so plain gradient descent slows down and settles smoothly without needing a decaying learning rate. The curvature is a constant 2/n2/n, so any ill-conditioning in training comes from the network, not from the loss. Many write 12r2\tfrac12 r^2 instead of r2r^2 so the gradient is simply y^−y\hat y-y; that only rescales the learning rate.

Properties

Range
[0,∞)[0,\infty); units are the target's units squared (report RMSE=MSE\mathrm{RMSE}=\sqrt{\mathrm{MSE}} to get back to the original units)
Minimizer
the mean of the targets; for a model, the conditional mean E[y∣x]\mathbb E[y\mid x]
Noise model
Gaussian with fixed variance: MSE\mathrm{MSE} is 2σ2×2\sigma^2\times the NLL plus a constant
Smoothness
infinitely differentiable, convex in y^\hat y, constant curvature 2/n2/n
Influence of one point
unbounded: the gradient grows linearly with the residual
PyTorch
nn.MSELoss(), F.mse_loss; nn.GaussianNLLLoss when the variance is predicted too

Why the mean: the minimizer of squared error

Suppose the model can only output one number cc for a set of targets (or, for a flexible network, one number per input xx). Expand the average squared error around the mean yˉ\bar y:

1n∑i(yi−c)2=(c−yˉ)2+1n∑i(yi−yˉ)2⏟s2⟹E[(Y−c)2∣x]=(c−E[Y∣x])2+Var⁡(Y∣x)\frac1n\sum_i (y_i-c)^2=(c-\bar y)^2+\underbrace{\frac1n\sum_i (y_i-\bar y)^2}_{s^2}\qquad\Longrightarrow\qquad \mathbb E\big[(Y-c)^2\mid x\big]=\big(c-\mathbb E[Y\mid x]\big)^2+\operatorname{Var}(Y\mid x)
yˉ\bar y
the sample mean; the unique minimizer
s2s^2
the variance of the targets: the floor no prediction can go below
Var⁡(Y∣x)\operatorname{Var}(Y\mid x)
irreducible noise; a perfect model still pays it

The cross term vanishes because residuals around the mean sum to zero. So L(c)L(c) is a parabola with its vertex at the mean and its floor at the variance. A network trained with MSE on enough data learns y^(x)≈E[y∣x]\hat y(x)\approx\mathbb E[y\mid x]: the average of everything that could follow xx, not the most likely value and not the median.

targets yᵢ, prediction c, residuals; bump = N(c, σ=1), −log of it = loss02468101214cL(c) = mean (yᵢ − c)² = (c − ȳ)² + s²0481216ȳ₇ȳ = 2.50c = 6.00 · MSE = 13.055 · dL/dc = 2(c − ȳ) = 7.00 · floor s² = 0.805NLL(σ=1) = MSE/2 + 0.919 = 7.446 · y₈ share of loss 11.7%
c − ȳ shrinks ×0.50 per step
fig.Eight targets and one constant prediction c. The parabola L(c) = (c − ȳ)² + s² has its vertex at the mean (2.50) and its floor at the variance (0.805). At the start, c = 6, the MSE is 13.055 and the slope 2(c − ȳ) = 7.00. Each GD step with η = 0.25 halves the distance to the mean (6 → 4.25 → 3.375 …), η = 0.5 lands on it in one step and η above 1 diverges. Now drag the red outlier y₈ from 2.5 to 12.5: the mean jumps to 3.75, the floor to 11.74, and at the new optimum that one point supplies 81.5% of the squared error. The bump is the Gaussian N(c, 1) whose negative log is the loss.

Why squares: MSE is the Gaussian negative log-likelihood

Model the target as the network output plus Gaussian noise, y=y^(x)+εy=\hat y(x)+\varepsilon with ε∼N(0,σ2)\varepsilon\sim\mathcal N(0,\sigma^2). The negative log-likelihood of one example is

−ln⁡p(y∣x)=(y−y^)22σ2+12ln⁡(2πσ2)-\ln p(y\mid x)=\frac{(y-\hat y)^2}{2\sigma^2}+\tfrac12\ln(2\pi\sigma^2)
σ2\sigma^2
noise variance; if it is a fixed constant it only rescales the loss
12ln⁡(2πσ2)\tfrac12\ln(2\pi\sigma^2)
does not depend on the network, so it has no gradient. At σ=1\sigma=1 it is 0.9190.919

With σ\sigma fixed, minimizing the summed NLL and minimizing MSE pick the same parameters, which is Gauss's argument. The choice of loss is therefore a modelling assumption: symmetric, light-tailed noise of the same size everywhere. Gaussian tails fall off like e−r2/2e^{-r^2/2}, so a residual of 6σ is e−18≈1.5×10−8e^{-18}\approx1.5\times10^{-8} times less likely than a perfect fit; the only way the model can make it less improbable is to move the fit toward it. That is the probabilistic reading of outlier sensitivity.

If the noise level varies with the input (heteroscedastic data), let the network predict a variance σ2(x)\sigma^2(x) as well and minimize the full NLL. The 12ln⁡σ2\tfrac12\ln\sigma^2 term then stops the model from claiming infinite uncertainty. PyTorch implements this as nn.GaussianNLLLoss; with the variance fixed at 1 it is exactly half the MSE.

Worked example

Four predictions against four targets; the last one is badly wrong. The residuals are the same ones used in the Loss functions overview.

iresidual r = y − ŷr²∂L/∂ŷᵢ = 2(ŷᵢ − yᵢ)/4share of loss
10.50.25−0.250.4%
2−110.51.4%
324−15.8%
4864−492.4%
y=(3, −0.5, 2, 7)y=(3,\,-0.5,\,2,\,7), y^=(2.5, 0.5, 0, −1)\hat y=(2.5,\,0.5,\,0,\,-1), n=4n=4 (checked numerically in Node)
  1. 1Sum of squares 0.25+1+4+64=69.250.25+1+4+64=69.25, so MSE=69.25/4=17.3125\mathrm{MSE}=69.25/4=17.3125 and RMSE=4.16\mathrm{RMSE}=4.16.
  2. 2The outlier's gradient −4-4 is 16 times the first example's and larger than the other three combined in magnitude (0.25+0.5+1=1.750.25+0.5+1=1.75). One update moves the shared weights mostly to please example 4.
  3. 3Gaussian NLL with σ=1\sigma=1: 12⋅17.3125+0.919=9.575\tfrac12\cdot17.3125+0.919=9.575 per example. nn.GaussianNLLLoss drops the constant and reports 8.6568.656.
  4. 4Compare (overview article): MAE =2.875=2.875 and Huberδ=1=2.406_{\delta=1}=2.406 on the same residuals, where the outlier's share drops to 70% and 78%.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3y = torch.tensor([3.0, -0.5, 2.0, 7.0])
4yhat = torch.tensor([2.5, 0.5, 0.0, -1.0], requires_grad=True)
5
6loss = nn.MSELoss()(yhat, y) # mean over every element
7loss.backward()
8print(loss.item()) # 17.3125
9print(yhat.grad) # tensor([-0.2500, 0.5000, -1.0000, -4.0000]) = 2(ŷ − y)/n
10F.mse_loss(yhat, y, reduction="sum") # 69.25
11
12# Same thing as a Gaussian likelihood; var may be predicted by the network (must be > 0)
13gnll = nn.GaussianNLLLoss() # 0.5 * (log var + (ŷ − y)² / var), constant omitted
14gnll(yhat, y, torch.ones(4)) # 8.6562 = MSE / 2
15
16# Shape bug: (N, 1) vs (N,) broadcasts to (N, N) and averages the wrong thing
17pred = yhat.detach().unsqueeze(1) # shape (4, 1)
18F.mse_loss(pred, y) # UserWarning: target size differs from input size
19F.mse_loss(pred.squeeze(1), y) # correct
MSE in PyTorch, its gradient, and the Gaussian NLL it comes from

Use when

  • +Smooth and convex in the prediction, with constant curvature: easy for every optimizer
  • +Gradient shrinks to 0 at the optimum, so training settles without a learning-rate schedule
  • +Statistically optimal (maximum likelihood) under Gaussian noise; estimates E[y∣x]\mathbb E[y\mid x]
  • +Closed-form solution for linear models (ordinary least squares)

Avoid when

  • −Outliers dominate: influence grows linearly with the residual
  • −Squared units: sensitive to target scaling
  • −Predicts the mean, which is blurry or unrepresentative for multimodal targets
  • −Poor gradient on sigmoid/softmax outputs; use cross-entropy for classification

Check yourself

Targets are 1, 2, 3 and 10. Which single constant prediction minimizes the MSE, and what is the minimum MSE?

Check yourself

You fit a constant cc by gradient descent on L(c)=1n∑i(yi−c)2L(c)=\frac1n\sum_i(y_i-c)^2. With which learning rate does one step land exactly on the mean from any starting point?

Remember

  • ✓MSE=1n∑(y−y^)2\mathrm{MSE}=\frac1n\sum(y-\hat y)^2, gradient 2n(y^−y)\frac2n(\hat y-y): linear in the error, constant curvature.
  • ✓Minimizer is the mean; a flexible model learns E[y∣x]\mathbb E[y\mid x] and still pays Var⁡(y∣x)\operatorname{Var}(y\mid x).
  • ✓MSE is the negative log-likelihood of Gaussian noise with fixed variance, up to scale and a constant; predict the variance too with GaussianNLLLoss.
  • ✓Unbounded influence: one large residual can supply most of the loss and the gradient. Use MAE or Huber when outliers are real.
  • ✓Standardize targets and check that prediction and target shapes match.
← back to Deep Learning roadmap