# Negative log-likelihood loss

> Pick a distribution for the target, let the network output its parameters, and minimize −log p(y | x). Categorical gives cross-entropy, Gaussian gives MSE or, with a learned variance, GaussianNLLLoss, and Poisson gives a loss for counts.

Source: https://www.mlfoundry.in/learn/negative-log-likelihood-loss  
Section: Deep Learning › Loss Functions  
Level: core · 12 min read · updated 2026-10-11

**Negative log-likelihood (NLL):** is the loss $-\ln p_\theta(y\mid x)$: the network outputs the parameters of a probability distribution over targets, and the loss is minus the log-probability (or log-density) that distribution assigns to the observed target. Averaged over a dataset, minimizing it is maximum-likelihood estimation.

NLL is less a single loss than a recipe for making losses. Choose a noise model, write down its log-density, negate it. The **Loss functions** overview used the recipe to show that MSE and cross-entropy are both NLLs. Here the recipe is the main subject: the general principle, PyTorch's `nn.NLLLoss` (which is specifically the categorical case and expects log-probabilities), and two NLLs that ordinary point losses cannot express: a Gaussian whose variance the network predicts, and a Poisson for counts.

Maximum likelihood was introduced and named by R. A. Fisher (1922), with earlier use by Gauss, who derived least squares from normally distributed errors in 1809. Under regularity conditions the maximum-likelihood estimator is consistent and asymptotically efficient, which is the classical argument for it. For neural networks, Nix and Weigend (1994) trained a network to output both a mean and a variance by minimizing the Gaussian NLL, the idea behind `nn.GaussianNLLLoss`.

## The principle

$$
\hat\theta=\arg\max_\theta\prod_{i=1}^n p_\theta(y_i\mid x_i)=\arg\min_\theta\ \underbrace{-\frac1n\sum_{i=1}^n\ln p_\theta(y_i\mid x_i)}_{\text{mean NLL}}
$$

- $p_\theta(y\mid x)$: the model: a distribution over $y$ whose parameters (mean, variance, rate, class probabilities…) are network outputs
- $\ln$: turns the product of many small numbers into a sum that does not underflow, without moving the optimum
- $\frac1n$: averaging makes the loss independent of dataset size; it is the cross-entropy between the empirical distribution and the model

Every observation contributes $-\ln p(y_i)$: zero only if the model puts probability 1 on it, large if the model thought it improbable. A model cannot cheat by assigning high probability everywhere, because a distribution has total mass 1. Raising the density in one place lowers it elsewhere, so NLL rewards putting mass where the data actually are.

Common output distributions and the loss each one yields (constants dropped where noted)

| target y | distribution | network outputs | −log p(y) up to constants | PyTorch |
| --- | --- | --- | --- | --- |
| binary | Bernoulli(p) | logit $z$, $p=\sigma(z)$ | $\operatorname{softplus}(z)-yz$ | `BCEWithLogitsLoss` |
| one of K classes | Categorical(p) | logits, $p=\operatorname{softmax}(z)$ | $-\ln p_y$ | `CrossEntropyLoss`, `NLLLoss` |
| real, fixed noise | $\mathcal N(\mu,\sigma^2)$, $\sigma$ const | $\mu$ | $(y-\mu)^2/2\sigma^2$ | `MSELoss` |
| real, learned noise | $\mathcal N(\mu,\sigma^2)$ | $\mu$, $\sigma^2>0$ | $\tfrac12[\ln\sigma^2+(y-\mu)^2/\sigma^2]$ | `GaussianNLLLoss` |
| real, heavy tails | Laplace$(\mu,b)$, $b$ const | $\mu$ | $\|y-\mu\|/b$ | `L1Loss` |
| count 0, 1, 2… | Poisson$(\lambda)$ | $\ln\lambda$ | $\lambda-y\ln\lambda$ | `PoissonNLLLoss` |

## nn.NLLLoss: the categorical case

In PyTorch, `nn.NLLLoss` is not the general recipe. It is the categorical NLL with the log-probabilities supplied by you: for input $\ell\in\mathbb R^{N\times K}$ and integer targets it returns $-\ell_{i,y_i}$, averaged. It does not take a log and it does not normalize. It is meant to follow `nn.LogSoftmax(dim=1)`; that pair is exactly `nn.CrossEntropyLoss`.

$$
\texttt{NLLLoss}(\ell,y)=-\frac{\sum_i w_{y_i}\,\ell_{i,y_i}}{\sum_i w_{y_i}},\qquad \texttt{CrossEntropyLoss}(z,y)=\texttt{NLLLoss}(\texttt{log\_softmax}(z),y)
$$

- $\ell_{i,k}$: log-probability of class $k$ for example $i$ (must already be normalized)
- $w_k$: optional class `weight`; all 1 by default, giving a plain mean

> **NLLLoss on the wrong input:** `NLLLoss` trusts its input to be log-probabilities. For logits $(2.0,\,0.5,\,-1.0)$ and target class 1, the correct value is $1.7413$. Feed it softmax probabilities and it returns $-0.1753$ (just $-p_1$); feed it raw logits and it returns $-0.5$. Both are negative, which a true NLL over a discrete distribution can never be, and training on them pushes the wrong quantity. Use `log_softmax`, or use `CrossEntropyLoss` on logits.

Separate `log_softmax` + `NLLLoss` is useful when the log-probabilities are needed elsewhere (beam search, sampling, a CTC or CRF layer) or when the model's output is a log-probability by construction, for example a mixture whose log-likelihood is computed with `logsumexp`.

> Figure: Eight observations and a model distribution. Each orange stem is the density the model gives one observation; the loss is the mean of −log of those heights. Gaussian tab: the starting model N(2, 1) has mean NLL 2.0639 and ∂/∂μ = −1.000. 'Fit by MLE' moves to μ = ȳ = 3.00 and σ² = 1.29 (the 1/n variance), where both gradients vanish and the NLL is 1.5463; nn.GaussianNLLLoss reports 0.6273 because it drops ½log 2π = 0.9189. With 'σ fixed at 1' the NLL is exactly ½·MSE + 0.919 (1.5639 at μ = 3): least squares. Overconfidence is expensive: σ = 0.5 at μ = 3 gives 2.81 (the readout shows 2.79 at the slider step nearest σ = 0.50). Poisson tab: counts 0,1,1,2,2,2,3,5 have MLE λ = 2.00 and NLL 1.6960.

## Gaussian NLL with a learned variance

MSE assumes every target has the same noise. Often it does not: some inputs are inherently harder to predict. Let the network output a mean $\mu(x)$ and a variance $\sigma^2(x)>0$ (usually through a softplus or an exponential), and minimize the Gaussian NLL:

$$
\ell=\tfrac12\Big[\ln\sigma^2+\frac{(y-\mu)^2}{\sigma^2}\Big]+\tfrac12\ln2\pi,\qquad \frac{\partial\ell}{\partial\mu}=\frac{\mu-y}{\sigma^2},\qquad\frac{\partial\ell}{\partial\sigma^2}=\frac12\Big[\frac1{\sigma^2}-\frac{(y-\mu)^2}{\sigma^4}\Big]
$$

- $\ln\sigma^2$: penalizes claiming large uncertainty
- $(y-\mu)^2/\sigma^2$: penalizes errors, discounted where the model admits uncertainty
- $\sigma^{2*}=(y-\mu)^2$: per example, the best variance equals the squared error; over a set, the mean squared error

The two terms balance: overclaiming confidence (small $\sigma^2$) makes the error term explode, and hedging (large $\sigma^2$) costs $\ln\sigma^2$. The result is a regression model that reports its own error bars. PyTorch's `nn.GaussianNLLLoss(full=False, eps=1e-6)` computes $\tfrac12[\ln\max(\sigma^2,\epsilon)+(y-\mu)^2/\max(\sigma^2,\epsilon)]$ and adds $\tfrac12\ln2\pi$ only with `full=True`. It takes the **variance**, not the standard deviation.

> **Learned variance can stall the mean:** The mean's gradient is divided by $\sigma^2$. Points the model finds hard get large predicted variance, which shrinks their gradient, so the model can settle for "uncertain and wrong" on exactly the difficult region. Seitzer et al. (2022) analyse this and propose β-NLL, which multiplies each term by a stop-gradient $\sigma^{2\beta}$. Common remedies are warming up with MSE before switching on the variance head, a variance floor, or β-NLL with $\beta\approx0.5$.

## Poisson NLL for counts

Counts (arrivals per hour, photons per pixel, clicks per day) are non-negative integers whose variance grows with their mean. A Poisson model with rate $\lambda(x)>0$ fits that. Writing the network's output as $u=\ln\lambda$ keeps the rate positive:

$$
-\ln\mathrm{Poisson}(y\mid\lambda)=\lambda-y\ln\lambda+\ln y!=e^u-yu+\ln y!,\qquad\frac{\partial\ell}{\partial u}=e^u-y=\lambda-y
$$

- $u=\ln\lambda$: log-rate, the network output (PyTorch `log_input=True`, the default)
- $\ln y!$: constant in the parameters; `full=True` adds a Stirling approximation of it for $y>1$
- $\lambda-y$: the familiar prediction-minus-target gradient, now in log-rate space

The gradient $\lambda-y$ on the natural parameter is not a coincidence. Bernoulli with a logit, categorical with logits, Gaussian with a mean and Poisson with a log-rate are all exponential-family distributions in their canonical parameter, and for every one of them the NLL gradient is mean minus observation. This is the same structure that generalized linear models use.

## Worked example

Eight observations y = (1.2, 2.0, 2.3, 2.9, 3.1, 3.4, 4.0, 5.1); mean NLL in nats (checked numerically in Node)

| model | μ | σ² | mean NLL | ∂/∂μ | ∂/∂σ² |
| --- | --- | --- | --- | --- | --- |
| start | 2.00 | 1.00 | 2.0639 | −1.000 | −0.645 |
| right mean, σ fixed | 3.00 | 1.00 | 1.5639 | 0 | −0.145 |
| MLE | 3.00 | 1.29 | 1.5463 | 0 | 0 |
| overconfident | 3.00 | 0.25 | 2.8058 | 0 | −8.320 |
| underconfident | 3.00 | 4.00 | 1.7733 | 0 | 0.085 |

1. $\bar y=3.0$ and $\frac1n\sum(y_i-\bar y)^2=1.29$: the MLE of the mean is the sample mean and of the variance the biased (divide-by-$n$) sample variance.
2. At $\sigma^2=1$ the NLL is $\tfrac12\mathrm{MSE}+\tfrac12\ln2\pi=\tfrac12(1.29)+0.9189=1.5639$: with a fixed variance, minimizing NLL is minimizing MSE.
3. Freeing $\sigma^2$ lowers the NLL to 1.5463. A negative $\partial/\partial\sigma^2$ means "increase the variance"; it is zero exactly at $\sigma^2=\mathrm{MSE}$.
4. `GaussianNLLLoss` at the MLE returns $1.5463-0.9189=0.6273$. For the counts $(0,1,1,2,2,2,3,5)$ the Poisson MLE is $\lambda=\bar y=2$, with mean NLL 1.6960 and `PoissonNLLLoss` value 0.6137 (no $\ln y!$ term).

> Figure: From the statistics chapter: the likelihood as a product of per-point densities. Drag μ and σ, or press 'Fit by MLE', and watch the log-likelihood peak at the sample mean and the 1/n standard deviation; the NLL loss is its negative.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

# categorical: NLLLoss expects log-probabilities
z = torch.tensor([[2.0, 0.5, -1.0]])
y = torch.tensor([1])
print(nn.NLLLoss()(F.log_softmax(z, dim=1), y))   # 1.7413
print(nn.CrossEntropyLoss()(z, y))                # 1.7413, same thing fused
print(nn.NLLLoss()(F.softmax(z, dim=1), y))       # -0.1753  <- wrong input, wrong loss

# Gaussian with a predicted variance (heteroscedastic regression)
target = torch.tensor([1.2, 2.0, 2.3, 2.9, 3.1, 3.4, 4.0, 5.1])
mean = torch.full_like(target, 3.0)
var = torch.full_like(target, 1.29)               # variance, not std; must be > 0
gnll = nn.GaussianNLLLoss()                       # full=False, eps=1e-6
print(gnll(mean, target, var))                    # 0.6273
print(nn.GaussianNLLLoss(full=True)(mean, target, var))   # 1.5463 = 0.6273 + 0.5*log(2*pi)

# a two-headed model: softplus keeps the variance positive
class MeanVar(nn.Module):
    def __init__(self, d):
        super().__init__()
        self.body = nn.Sequential(nn.Linear(d, 64), nn.ReLU())
        self.mu, self.raw_var = nn.Linear(64, 1), nn.Linear(64, 1)
    def forward(self, x):
        h = self.body(x)
        return self.mu(h).squeeze(-1), F.softplus(self.raw_var(h)).squeeze(-1) + 1e-6

# Poisson: the network outputs log-rates
counts = torch.tensor([0., 1., 1., 2., 2., 2., 3., 5.])
log_rate = torch.full_like(counts, math.log(2.0))
print(nn.PoissonNLLLoss()(log_rate, counts))      # 0.6137 = mean(exp(u) - y*u)
```

Categorical, Gaussian and Poisson NLL in PyTorch

> **The loss is the noise model:** Every NLL encodes an assumption about how targets scatter around the prediction. If residuals have outliers, a Gaussian is the wrong model and a Laplace or Student-t NLL fits better; if targets are counts, a Gaussian wastes probability on negative and fractional values. Choosing a loss is choosing $p(y\mid x)$, and the fitted model is only as good as that choice.

> **Discrete vs continuous NLL:** For discrete targets $-\ln p\ge0$, so NLL is non-negative. For continuous targets $p$ is a density, which can exceed 1, so a Gaussian NLL can be negative (a very confident correct prediction with $\sigma^2<1/2\pi$). Negative values are not a bug there, but they do mean you cannot read the number as "nats of surprise" the way you can for classification.

**Strengths**

- A principled, general recipe: one loss per noise model, with known statistical properties (consistency, efficiency)
- Can learn uncertainty (variance, rate, mixture weights), not just a point prediction
- Gradients on canonical parameters are simply mean minus observation
- Losses from different models are comparable as log-likelihoods on held-out data

**Limitations**

- Only as good as the assumed distribution; a wrong noise model gives a confident wrong fit
- Learned variances can collapse or grow to dodge hard examples; needs floors, warm-up or β-NLL
- PyTorch `NLLLoss` silently accepts non-log-probabilities and returns meaningless values
- Unbounded: one very improbable observation can dominate the mean

**Question:** A regression model outputs (μ, σ²) and is trained with GaussianNLLLoss. On one example its error is |y − μ| = 2. Holding μ fixed, what variance minimizes that example's loss?

- A. σ² = 2
- B. σ² = 4
- C. σ² = 1
- D. σ² → ∞

**Answer:** B. σ² = 4 — $\partial\ell/\partial\sigma^2=\tfrac12[1/\sigma^2-(y-\mu)^2/\sigma^4]=0$ gives $\sigma^2=(y-\mu)^2=4$. Larger variance pays the $\ln\sigma^2$ penalty, smaller variance pays the error term.

**Question:** Why does nn.NLLLoss(F.softmax(z, dim=1), y) produce negative losses?

- A. Softmax outputs can be negative
- B. NLLLoss returns −input[y] without taking a log, so it computes −p_y, which lies in (−1, 0)
- C. The class weights are wrong
- D. NLLLoss always returns negative values

**Answer:** B. NLLLoss returns −input[y] without taking a log, so it computes −p_y, which lies in (−1, 0) — `NLLLoss` only gathers and negates. It expects log-probabilities ($\le0$), which give non-negative losses. Given probabilities in $(0,1)$ it returns $-p_y$; minimizing that maximizes $p_y$ linearly, with none of cross-entropy's gradient behaviour. Use `log_softmax`.

**Key points**

- NLL $=-\ln p_\theta(y\mid x)$: choose a distribution, let the network output its parameters; minimizing the mean is maximum likelihood.
- Categorical → cross-entropy; Gaussian with fixed $\sigma$ → MSE; Laplace → MAE; Bernoulli → BCE; Poisson → $\lambda-y\ln\lambda$.
- PyTorch `NLLLoss` is the categorical case on log-probabilities: `log_softmax` + `NLLLoss` = `CrossEntropyLoss`.
- `GaussianNLLLoss` learns a variance: $\tfrac12[\ln\sigma^2+(y-\mu)^2/\sigma^2]$, optimal $\sigma^2=$ squared error; watch for variance-induced gradient shrinkage.
- For canonical parameters (logit, logits, mean, log-rate) the gradient is prediction minus target.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
