Loss functions

MSE for regression and cross-entropy for classification are both negative log-likelihoods.

8 min read823 wordscore0 viewsupdated 2026-10-09

Definition

A loss function maps a prediction y^\hat y and a target yy to one non-negative number that says how wrong the prediction is. Training minimizes its average over the data, 1n∑iℓ(y^i,yi)\frac1n\sum_i \ell(\hat y_i, y_i), and its gradient is the only signal backpropagation ever sends into the network.

The loss decides what "good" means. Two models with the same architecture can learn different things only because one is trained with squared error and the other with absolute error.

It also decides the gradient. A loss that is flat where the model is badly wrong gives almost no learning signal exactly when it is needed most. Much of the craft of choosing a loss is about the shape of its gradient, not its value.

Regression losses

Write the residual as r=y−y^r = y-\hat y. The three standard choices differ only in how they grow with ∣r∣|r|:

MSE: r2MAE: ∣r∣Huberδ:{12r2∣r∣≤δδ (∣r∣−12δ)∣r∣>δ\text{MSE: } r^2 \qquad \text{MAE: } |r| \qquad \text{Huber}_\delta: \begin{cases}\tfrac12 r^2 & |r|\le\delta\\ \delta\,(|r|-\tfrac12\delta) & |r|>\delta\end{cases}
rr
residual y−y^y-\hat y for one example
δ\delta
Huber threshold: quadratic inside, linear outside

MSE has gradient 2r2r: big errors get big pushes, so a single outlier can dominate a batch. Its minimizer over a set of targets is the mean.

MAE has gradient sign⁡(r)\operatorname{sign}(r): every example pushes with the same force, so outliers matter less. Its minimizer is the median. The gradient does not shrink near the optimum, so it tends to jitter around the answer unless the learning rate decays.

Huber is MSE for small residuals and MAE for large ones. Its gradient is rr clipped to [−δ,δ][-\delta,\delta], which is smooth near zero and bounded for outliers. In PyTorch it is nn.HuberLoss(delta=…). nn.SmoothL1Loss is the same curve divided by δ\delta.

Classification losses

For a binary label y∈{0,1}y\in\{0,1\} the network outputs a logit zz and a probability p=σ(z)p=\sigma(z). Binary cross-entropy is

ℓBCE=−[yln⁡p+(1−y)ln⁡(1−p)]\ell_{\text{BCE}} = -\bigl[y\ln p + (1-y)\ln(1-p)\bigr]
p=σ(z)p=\sigma(z)
predicted probability of class 1; σ(z)=1/(1+e−z)\sigma(z)=1/(1+e^{-z})
yy
true label, 0 or 1

For KK classes the network outputs logits z∈RKz\in\mathbb{R}^K, softmax turns them into probabilities, and categorical cross-entropy is minus the log-probability of the true class cc:

pk=ezk∑jezj,ℓCE=−∑kykln⁡pk=−ln⁡pc=log⁡∑jezj−zcp_k=\frac{e^{z_k}}{\sum_j e^{z_j}},\qquad \ell_{\text{CE}}=-\sum_k y_k\ln p_k=-\ln p_c=\log\sum_j e^{z_j}-z_c
zkz_k
logit (raw score) for class kk
yky_k
one-hot target: 1 for the true class cc, 0 otherwise
pcp_c
predicted probability of the true class

Cross-entropy is unbounded: as pc→0p_c\to0 the loss goes to infinity. A confident wrong answer (pc=0.01p_c=0.01) costs 4.64.6 nats, while a hesitant one (pc=0.4p_c=0.4) costs 0.920.92.

Both are negative log-likelihoods

Treat the network as a probability model p(y∣x;θ)p(y\mid x;\theta) and fit it by maximum likelihood. Maximizing ∏ip(yi∣xi)\prod_i p(y_i\mid x_i) is the same as minimizing −∑iln⁡p(yi∣xi)-\sum_i \ln p(y_i\mid x_i). The loss is what you get once you pick a distribution for yy.

Gaussian noise gives MSE. Assume y∼N(y^,σ2)y\sim\mathcal N(\hat y,\sigma^2) with fixed σ\sigma:

−ln⁡p(y∣x)=(y−y^)22σ2+12ln⁡(2πσ2)-\ln p(y\mid x)=\frac{(y-\hat y)^2}{2\sigma^2}+\tfrac12\ln(2\pi\sigma^2)
y^\hat y
network output, used as the mean of the Gaussian
σ2\sigma^2
noise variance; a constant, so it only rescales the loss

The second term does not depend on θ\theta and the factor 1/2σ21/2\sigma^2 only rescales, so minimizing NLL is minimizing squared error. A Laplace noise model gives MAE in the same way.

Bernoulli labels give BCE. Assume y∼Bernoulli(p)y\sim\mathrm{Bernoulli}(p), so p(y)=py(1−p)1−yp(y)=p^y(1-p)^{1-y}. Taking −ln⁡-\ln gives exactly −[yln⁡p+(1−y)ln⁡(1−p)]-[y\ln p+(1-y)\ln(1-p)]. A categorical distribution gives categorical cross-entropy.

The gradient: p − y

Softmax and cross-entropy combine into a remarkably simple gradient with respect to the logits:

∂ℓCE∂zk=pk−yk\frac{\partial \ell_{\text{CE}}}{\partial z_k}=p_k-y_k
pk−ykp_k-y_k
predicted minus target probability, for each class

Differentiate log⁡∑jezj−zc\log\sum_j e^{z_j}-z_c: the first term gives pkp_k, the second gives 11 for k=ck=c. The same holds for sigmoid + BCE: ∂ℓ/∂z=p−y\partial\ell/\partial z=p-y. The error signal is bounded by 1 and never saturates while the answer is wrong. Squared error on a sigmoid output has gradient 2(p−y) p(1−p)2(p-y)\,p(1-p), which goes to zero as a wrong prediction becomes confident. That is why classifiers are trained with cross-entropy.

loss ℓ(r)01234−3−2−10123residual r = y − ŷgradient ∂ℓ/∂r−3−2−10123−3−2−10123residual r— MSE r² — MAE |r| ┄ Huber δ
MSE 2.250 · grad 3.000
MAE 1.500 · grad 1.000
Huber 1.000 · grad 1.000
fig.Regression: drag the residual and watch MSE's gradient grow without bound while MAE's stays at ±1; shrink Huber's δ to see it move from MSE-like to MAE-like. Classification: slide p(true) toward 0 and compare the gradients. Cross-entropy's approaches 1, while squared error's collapses to 0 exactly when the model is confidently wrong. The dashed marker shows the ln K loss of a uniform prediction.

Worked example

classlogit zsoftmax ptarget ygradient p − y
1 (true)2.00.6591−0.341
21.00.24200.242
30.10.09900.099
Three classes, logits z=(2.0, 1.0, 0.1)z=(2.0,\,1.0,\,0.1), true class 1. log⁡∑jezj=2.417\log\sum_j e^{z_j}=2.417.
  1. 1Loss =2.417−2.0=0.417=2.417-2.0=0.417 nats, i.e. −ln⁡0.659-\ln 0.659.
  2. 2If class 3 were the true one, the loss would be 2.417−0.1=2.3172.417-0.1=2.317: the confident wrong answer costs 5.6 times more.
  3. 3The gradient raises the true logit (negative entry) and lowers the others in proportion to how much probability they stole. The entries sum to 0.
  4. 4Regression, residuals (0.5, −1, 2, 8)(0.5,\,-1,\,2,\,8): MSE =17.31=17.31, MAE =2.875=2.875, Huberδ=1=2.406_{\delta=1}=2.406. The outlier r=8r=8 supplies 92% of the squared error but only 78% of the Huber loss and 70% of the absolute error.

In practice: pass logits, not probabilities

Computing ln⁡σ(z)\ln\sigma(z) naively fails for large ∣z∣|z|: σ(−1000)\sigma(-1000) underflows to 0 and the log becomes −∞-\infty. Stable implementations work on the logit directly, using ln⁡σ(z)=−log⁡(1+e−z)\ln\sigma(z)=-\log(1+e^{-z}) and the log-sum-exp trick (subtract max⁡jzj\max_j z_j before exponentiating).

python
1import torch, torch.nn as nn
2
3logits = torch.tensor([[2.0, 1.0, 0.1]])
4target = torch.tensor([0]) # index of the true class (class 1 above), not one-hot
5nn.CrossEntropyLoss()(logits, target) # 0.4170: log-softmax + NLL, fused
6
7# binary: one logit per example
8z, y = torch.tensor([3.0]), torch.tensor([1.0])
9nn.BCEWithLogitsLoss()(z, y) # 0.0486 = -ln σ(3)
10
11nn.MSELoss(), nn.L1Loss(), nn.HuberLoss(delta=1.0) # regression

Check yourself

A 10-class classifier reports a loss of 2.30 on its first batch. What does that indicate?

Check yourself

Why is squared error a poor loss for a sigmoid classifier?

Remember

  • ✓Choosing a loss is choosing a noise model: Gaussian → MSE, Laplace → MAE, Bernoulli/categorical → cross-entropy.
  • ✓MSE targets the mean and is outlier-sensitive; MAE targets the median; Huber is quadratic near 0 and linear beyond δ.
  • ✓Softmax + cross-entropy has gradient p − y w.r.t. the logits: bounded and non-saturating.
  • ✓Feed logits to CrossEntropyLoss / BCEWithLogitsLoss; they are numerically stable.
  • ✓Expect a loss of about ln K at initialization.
← back to Deep Learning roadmap