# Loss functions

> MSE for regression and cross-entropy for classification are both negative log-likelihoods.

Source: https://www.mlfoundry.in/learn/loss-functions  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 8 min read · updated 2026-10-09

**A loss function:** maps a prediction $\hat y$ and a target $y$ to one non-negative number that says how wrong the prediction is. Training minimizes its average over the data, $\frac1n\sum_i \ell(\hat y_i, y_i)$, and its gradient is the only signal backpropagation ever sends into the network.

The loss decides what "good" means. Two models with the same architecture can learn different things only because one is trained with squared error and the other with absolute error.

It also decides the **gradient**. A loss that is flat where the model is badly wrong gives almost no learning signal exactly when it is needed most. Much of the craft of choosing a loss is about the shape of its gradient, not its value.

## Regression losses

Write the residual as $r = y-\hat y$. The three standard choices differ only in how they grow with $|r|$:

$$
\text{MSE: } r^2 \qquad \text{MAE: } |r| \qquad \text{Huber}_\delta: \begin{cases}\tfrac12 r^2 & |r|\le\delta\\ \delta\,(|r|-\tfrac12\delta) & |r|>\delta\end{cases}
$$

- $r$: residual $y-\hat y$ for one example
- $\delta$: Huber threshold: quadratic inside, linear outside

**MSE** has gradient $2r$: big errors get big pushes, so a single outlier can dominate a batch. Its minimizer over a set of targets is the **mean**.

**MAE** has gradient $\operatorname{sign}(r)$: every example pushes with the same force, so outliers matter less. Its minimizer is the **median**. The gradient does not shrink near the optimum, so it tends to jitter around the answer unless the learning rate decays.

**Huber** is MSE for small residuals and MAE for large ones. Its gradient is $r$ clipped to $[-\delta,\delta]$, which is smooth near zero and bounded for outliers. In PyTorch it is `nn.HuberLoss(delta=…)`. `nn.SmoothL1Loss` is the same curve divided by $\delta$.

## Classification losses

For a binary label $y\in\{0,1\}$ the network outputs a logit $z$ and a probability $p=\sigma(z)$. **Binary cross-entropy** is

$$
\ell_{\text{BCE}} = -\bigl[y\ln p + (1-y)\ln(1-p)\bigr]
$$

- $p=\sigma(z)$: predicted probability of class 1; $\sigma(z)=1/(1+e^{-z})$
- $y$: true label, 0 or 1

For $K$ classes the network outputs logits $z\in\mathbb{R}^K$, softmax turns them into probabilities, and **categorical cross-entropy** is minus the log-probability of the true class $c$:

$$
p_k=\frac{e^{z_k}}{\sum_j e^{z_j}},\qquad \ell_{\text{CE}}=-\sum_k y_k\ln p_k=-\ln p_c=\log\sum_j e^{z_j}-z_c
$$

- $z_k$: logit (raw score) for class $k$
- $y_k$: one-hot target: 1 for the true class $c$, 0 otherwise
- $p_c$: predicted probability of the true class

Cross-entropy is unbounded: as $p_c\to0$ the loss goes to infinity. A confident wrong answer ($p_c=0.01$) costs $4.6$ nats, while a hesitant one ($p_c=0.4$) costs $0.92$.

## Both are negative log-likelihoods

Treat the network as a probability model $p(y\mid x;\theta)$ and fit it by maximum likelihood. Maximizing $\prod_i p(y_i\mid x_i)$ is the same as minimizing $-\sum_i \ln p(y_i\mid x_i)$. The loss is what you get once you pick a distribution for $y$.

**Gaussian noise gives MSE.** Assume $y\sim\mathcal N(\hat y,\sigma^2)$ with fixed $\sigma$:

$$
-\ln p(y\mid x)=\frac{(y-\hat y)^2}{2\sigma^2}+\tfrac12\ln(2\pi\sigma^2)
$$

- $\hat y$: network output, used as the mean of the Gaussian
- $\sigma^2$: noise variance; a constant, so it only rescales the loss

The second term does not depend on $\theta$ and the factor $1/2\sigma^2$ only rescales, so minimizing NLL is minimizing squared error. A Laplace noise model gives MAE in the same way.

**Bernoulli labels give BCE.** Assume $y\sim\mathrm{Bernoulli}(p)$, so $p(y)=p^y(1-p)^{1-y}$. Taking $-\ln$ gives exactly $-[y\ln p+(1-y)\ln(1-p)]$. A categorical distribution gives categorical cross-entropy.

> **Why this matters:** The loss is a modelling assumption. MSE assumes symmetric, light-tailed noise; if your targets have outliers, the Gaussian is wrong and Huber or MAE matches the data better. Probabilistic outputs trained with cross-entropy are calibrated estimates of $p(y\mid x)$ exactly because they are maximum-likelihood fits.

## The gradient: p − y

Softmax and cross-entropy combine into a remarkably simple gradient with respect to the logits:

$$
\frac{\partial \ell_{\text{CE}}}{\partial z_k}=p_k-y_k
$$

- $p_k-y_k$: predicted minus target probability, for each class

Differentiate $\log\sum_j e^{z_j}-z_c$: the first term gives $p_k$, the second gives $1$ for $k=c$. The same holds for sigmoid + BCE: $\partial\ell/\partial z=p-y$. The error signal is bounded by 1 and never saturates while the answer is wrong. Squared error on a sigmoid output has gradient $2(p-y)\,p(1-p)$, which goes to zero as a wrong prediction becomes confident. That is why classifiers are trained with cross-entropy.

> Figure: Regression: drag the residual and watch MSE's gradient grow without bound while MAE's stays at ±1; shrink Huber's δ to see it move from MSE-like to MAE-like. Classification: slide p(true) toward 0 and compare the gradients. Cross-entropy's approaches 1, while squared error's collapses to 0 exactly when the model is confidently wrong. The dashed marker shows the ln K loss of a uniform prediction.

## Worked example

Three classes, logits $z=(2.0,\,1.0,\,0.1)$, true class 1. $\log\sum_j e^{z_j}=2.417$.

| class | logit z | softmax p | target y | gradient p − y |
| --- | --- | --- | --- | --- |
| 1 (true) | 2.0 | 0.659 | 1 | −0.341 |
| 2 | 1.0 | 0.242 | 0 | 0.242 |
| 3 | 0.1 | 0.099 | 0 | 0.099 |

1. Loss $=2.417-2.0=0.417$ nats, i.e. $-\ln 0.659$.
2. If class 3 were the true one, the loss would be $2.417-0.1=2.317$: the confident wrong answer costs 5.6 times more.
3. The gradient raises the true logit (negative entry) and lowers the others in proportion to how much probability they stole. The entries sum to 0.
4. Regression, residuals $(0.5,\,-1,\,2,\,8)$: MSE $=17.31$, MAE $=2.875$, Huber$_{\delta=1}=2.406$. The outlier $r=8$ supplies 92% of the squared error but only 78% of the Huber loss and 70% of the absolute error.

## In practice: pass logits, not probabilities

Computing $\ln\sigma(z)$ naively fails for large $|z|$: $\sigma(-1000)$ underflows to 0 and the log becomes $-\infty$. Stable implementations work on the logit directly, using $\ln\sigma(z)=-\log(1+e^{-z})$ and the log-sum-exp trick (subtract $\max_j z_j$ before exponentiating).

```python
import torch, torch.nn as nn

logits = torch.tensor([[2.0, 1.0, 0.1]])
target = torch.tensor([0])                    # index of the true class (class 1 above), not one-hot
nn.CrossEntropyLoss()(logits, target)         # 0.4170: log-softmax + NLL, fused

# binary: one logit per example
z, y = torch.tensor([3.0]), torch.tensor([1.0])
nn.BCEWithLogitsLoss()(z, y)                  # 0.0486 = -ln σ(3)

nn.MSELoss(), nn.L1Loss(), nn.HuberLoss(delta=1.0)   # regression
```

> **Softmax twice:** `nn.CrossEntropyLoss` applies log-softmax itself. Putting `nn.Softmax` at the end of the model and then calling it squashes the probabilities a second time: training still runs, but gradients shrink and accuracy suffers. The same goes for `nn.Sigmoid` followed by `BCEWithLogitsLoss`. Keep the model's output as raw logits.

> **Sanity check at initialization:** A freshly initialized classifier has logits near 0, so softmax is close to uniform and $p_c\approx1/K$. The first loss should be $\approx\ln K$: $0.693$ for 2 classes, $2.303$ for 10, $6.908$ for 1000. A much larger value means the output layer is initialized too large or the labels are wrong.

**Question:** A 10-class classifier reports a loss of 2.30 on its first batch. What does that indicate?

- A. It has already learned something
- B. Its predictions are roughly uniform, as expected at initialization
- C. The labels are shuffled
- D. The learning rate is too high

**Answer:** B. Its predictions are roughly uniform, as expected at initialization — $\ln 10=2.303$, the cross-entropy of a uniform prediction $p_c=0.1$. That is the expected starting point; anything far above it suggests a bug.

**Question:** Why is squared error a poor loss for a sigmoid classifier?

- A. It is not differentiable
- B. Its gradient w.r.t. the logit vanishes when the model is confidently wrong
- C. It cannot be minimized
- D. It always overfits

**Answer:** B. Its gradient w.r.t. the logit vanishes when the model is confidently wrong — $\partial(p-y)^2/\partial z=2(p-y)\,p(1-p)$. If $y=1$ and $p\to0$, the factor $p$ drives the gradient to 0. Cross-entropy's gradient is $p-y\to-1$, a full-strength signal.

**Key points**

- Choosing a loss is choosing a noise model: Gaussian → MSE, Laplace → MAE, Bernoulli/categorical → cross-entropy.
- MSE targets the mean and is outlier-sensitive; MAE targets the median; Huber is quadratic near 0 and linear beyond δ.
- Softmax + cross-entropy has gradient p − y w.r.t. the logits: bounded and non-saturating.
- Feed logits to `CrossEntropyLoss` / `BCEWithLogitsLoss`; they are numerically stable.
- Expect a loss of about ln K at initialization.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
