Definition
A loss function maps a prediction and a target to one non-negative number that says how wrong the prediction is. Training minimizes its average over the data, , and its gradient is the only signal backpropagation ever sends into the network.
The loss decides what "good" means. Two models with the same architecture can learn different things only because one is trained with squared error and the other with absolute error.
It also decides the gradient. A loss that is flat where the model is badly wrong gives almost no learning signal exactly when it is needed most. Much of the craft of choosing a loss is about the shape of its gradient, not its value.
Regression losses
Write the residual as . The three standard choices differ only in how they grow with :
- residual for one example
- Huber threshold: quadratic inside, linear outside
MSE has gradient : big errors get big pushes, so a single outlier can dominate a batch. Its minimizer over a set of targets is the mean.
MAE has gradient : every example pushes with the same force, so outliers matter less. Its minimizer is the median. The gradient does not shrink near the optimum, so it tends to jitter around the answer unless the learning rate decays.
Huber is MSE for small residuals and MAE for large ones. Its gradient is clipped to , which is smooth near zero and bounded for outliers. In PyTorch it is nn.HuberLoss(delta=…). nn.SmoothL1Loss is the same curve divided by .
Classification losses
For a binary label the network outputs a logit and a probability . Binary cross-entropy is
- predicted probability of class 1;
- true label, 0 or 1
For classes the network outputs logits , softmax turns them into probabilities, and categorical cross-entropy is minus the log-probability of the true class :
- logit (raw score) for class
- one-hot target: 1 for the true class , 0 otherwise
- predicted probability of the true class
Cross-entropy is unbounded: as the loss goes to infinity. A confident wrong answer () costs nats, while a hesitant one () costs .
Both are negative log-likelihoods
Treat the network as a probability model and fit it by maximum likelihood. Maximizing is the same as minimizing . The loss is what you get once you pick a distribution for .
Gaussian noise gives MSE. Assume with fixed :
- network output, used as the mean of the Gaussian
- noise variance; a constant, so it only rescales the loss
The second term does not depend on and the factor only rescales, so minimizing NLL is minimizing squared error. A Laplace noise model gives MAE in the same way.
Bernoulli labels give BCE. Assume , so . Taking gives exactly . A categorical distribution gives categorical cross-entropy.
The gradient: p − y
Softmax and cross-entropy combine into a remarkably simple gradient with respect to the logits:
- predicted minus target probability, for each class
Differentiate : the first term gives , the second gives for . The same holds for sigmoid + BCE: . The error signal is bounded by 1 and never saturates while the answer is wrong. Squared error on a sigmoid output has gradient , which goes to zero as a wrong prediction becomes confident. That is why classifiers are trained with cross-entropy.
Worked example
| class | logit z | softmax p | target y | gradient p − y |
|---|---|---|---|---|
| 1 (true) | 2.0 | 0.659 | 1 | −0.341 |
| 2 | 1.0 | 0.242 | 0 | 0.242 |
| 3 | 0.1 | 0.099 | 0 | 0.099 |
- 1Loss nats, i.e. .
- 2If class 3 were the true one, the loss would be : the confident wrong answer costs 5.6 times more.
- 3The gradient raises the true logit (negative entry) and lowers the others in proportion to how much probability they stole. The entries sum to 0.
- 4Regression, residuals : MSE , MAE , Huber. The outlier supplies 92% of the squared error but only 78% of the Huber loss and 70% of the absolute error.
In practice: pass logits, not probabilities
Computing naively fails for large : underflows to 0 and the log becomes . Stable implementations work on the logit directly, using and the log-sum-exp trick (subtract before exponentiating).
1import torch, torch.nn as nn23logits = torch.tensor([[2.0, 1.0, 0.1]])4target = torch.tensor([0]) # index of the true class (class 1 above), not one-hot5nn.CrossEntropyLoss()(logits, target) # 0.4170: log-softmax + NLL, fused67# binary: one logit per example8z, y = torch.tensor([3.0]), torch.tensor([1.0])9nn.BCEWithLogitsLoss()(z, y) # 0.0486 = -ln σ(3)1011nn.MSELoss(), nn.L1Loss(), nn.HuberLoss(delta=1.0) # regressionCheck yourself
A 10-class classifier reports a loss of 2.30 on its first batch. What does that indicate?
Check yourself
Why is squared error a poor loss for a sigmoid classifier?
Remember
- ✓Choosing a loss is choosing a noise model: Gaussian → MSE, Laplace → MAE, Bernoulli/categorical → cross-entropy.
- ✓MSE targets the mean and is outlier-sensitive; MAE targets the median; Huber is quadratic near 0 and linear beyond δ.
- ✓Softmax + cross-entropy has gradient p − y w.r.t. the logits: bounded and non-saturating.
- ✓Feed logits to
CrossEntropyLoss/BCEWithLogitsLoss; they are numerically stable. - ✓Expect a loss of about ln K at initialization.