Definition
Negative log-likelihood (NLL) is the loss : the network outputs the parameters of a probability distribution over targets, and the loss is minus the log-probability (or log-density) that distribution assigns to the observed target. Averaged over a dataset, minimizing it is maximum-likelihood estimation.
NLL is less a single loss than a recipe for making losses. Choose a noise model, write down its log-density, negate it. The Loss functions overview used the recipe to show that MSE and cross-entropy are both NLLs. Here the recipe is the main subject: the general principle, PyTorch's nn.NLLLoss (which is specifically the categorical case and expects log-probabilities), and two NLLs that ordinary point losses cannot express: a Gaussian whose variance the network predicts, and a Poisson for counts.
Maximum likelihood was introduced and named by R. A. Fisher (1922), with earlier use by Gauss, who derived least squares from normally distributed errors in 1809. Under regularity conditions the maximum-likelihood estimator is consistent and asymptotically efficient, which is the classical argument for it. For neural networks, Nix and Weigend (1994) trained a network to output both a mean and a variance by minimizing the Gaussian NLL, the idea behind nn.GaussianNLLLoss.
The principle
- the model: a distribution over whose parameters (mean, variance, rate, class probabilities…) are network outputs
- turns the product of many small numbers into a sum that does not underflow, without moving the optimum
- averaging makes the loss independent of dataset size; it is the cross-entropy between the empirical distribution and the model
Every observation contributes : zero only if the model puts probability 1 on it, large if the model thought it improbable. A model cannot cheat by assigning high probability everywhere, because a distribution has total mass 1. Raising the density in one place lowers it elsewhere, so NLL rewards putting mass where the data actually are.
| target y | distribution | network outputs | −log p(y) up to constants | PyTorch |
|---|---|---|---|---|
| binary | Bernoulli(p) | logit , | BCEWithLogitsLoss | |
| one of K classes | Categorical(p) | logits, | CrossEntropyLoss, NLLLoss | |
| real, fixed noise | , const | MSELoss | ||
| real, learned noise | , | GaussianNLLLoss | ||
| real, heavy tails | Laplace, const | L1Loss | ||
| count 0, 1, 2… | Poisson | PoissonNLLLoss |
nn.NLLLoss: the categorical case
In PyTorch, nn.NLLLoss is not the general recipe. It is the categorical NLL with the log-probabilities supplied by you: for input and integer targets it returns , averaged. It does not take a log and it does not normalize. It is meant to follow nn.LogSoftmax(dim=1); that pair is exactly nn.CrossEntropyLoss.
- log-probability of class for example (must already be normalized)
- optional class
weight; all 1 by default, giving a plain mean
Separate log_softmax + NLLLoss is useful when the log-probabilities are needed elsewhere (beam search, sampling, a CTC or CRF layer) or when the model's output is a log-probability by construction, for example a mixture whose log-likelihood is computed with logsumexp.
Gaussian NLL with a learned variance
MSE assumes every target has the same noise. Often it does not: some inputs are inherently harder to predict. Let the network output a mean and a variance (usually through a softplus or an exponential), and minimize the Gaussian NLL:
- penalizes claiming large uncertainty
- penalizes errors, discounted where the model admits uncertainty
- per example, the best variance equals the squared error; over a set, the mean squared error
The two terms balance: overclaiming confidence (small ) makes the error term explode, and hedging (large ) costs . The result is a regression model that reports its own error bars. PyTorch's nn.GaussianNLLLoss(full=False, eps=1e-6) computes and adds only with full=True. It takes the variance, not the standard deviation.
Poisson NLL for counts
Counts (arrivals per hour, photons per pixel, clicks per day) are non-negative integers whose variance grows with their mean. A Poisson model with rate fits that. Writing the network's output as keeps the rate positive:
- log-rate, the network output (PyTorch
log_input=True, the default) - constant in the parameters;
full=Trueadds a Stirling approximation of it for - the familiar prediction-minus-target gradient, now in log-rate space
The gradient on the natural parameter is not a coincidence. Bernoulli with a logit, categorical with logits, Gaussian with a mean and Poisson with a log-rate are all exponential-family distributions in their canonical parameter, and for every one of them the NLL gradient is mean minus observation. This is the same structure that generalized linear models use.
Worked example
| model | μ | σ² | mean NLL | ∂/∂μ | ∂/∂σ² |
|---|---|---|---|---|---|
| start | 2.00 | 1.00 | 2.0639 | −1.000 | −0.645 |
| right mean, σ fixed | 3.00 | 1.00 | 1.5639 | 0 | −0.145 |
| MLE | 3.00 | 1.29 | 1.5463 | 0 | 0 |
| overconfident | 3.00 | 0.25 | 2.8058 | 0 | −8.320 |
| underconfident | 3.00 | 4.00 | 1.7733 | 0 | 0.085 |
- 1 and : the MLE of the mean is the sample mean and of the variance the biased (divide-by-) sample variance.
- 2At the NLL is : with a fixed variance, minimizing NLL is minimizing MSE.
- 3Freeing lowers the NLL to 1.5463. A negative means "increase the variance"; it is zero exactly at .
- 4
GaussianNLLLossat the MLE returns . For the counts the Poisson MLE is , with mean NLL 1.6960 andPoissonNLLLossvalue 0.6137 (no term).
1import math, torch, torch.nn as nn, torch.nn.functional as F23# categorical: NLLLoss expects log-probabilities4z = torch.tensor([[2.0, 0.5, -1.0]])5y = torch.tensor([1])6print(nn.NLLLoss()(F.log_softmax(z, dim=1), y)) # 1.74137print(nn.CrossEntropyLoss()(z, y)) # 1.7413, same thing fused8print(nn.NLLLoss()(F.softmax(z, dim=1), y)) # -0.1753 <- wrong input, wrong loss910# Gaussian with a predicted variance (heteroscedastic regression)11target = torch.tensor([1.2, 2.0, 2.3, 2.9, 3.1, 3.4, 4.0, 5.1])12mean = torch.full_like(target, 3.0)13var = torch.full_like(target, 1.29) # variance, not std; must be > 014gnll = nn.GaussianNLLLoss() # full=False, eps=1e-615print(gnll(mean, target, var)) # 0.627316print(nn.GaussianNLLLoss(full=True)(mean, target, var)) # 1.5463 = 0.6273 + 0.5*log(2*pi)1718# a two-headed model: softplus keeps the variance positive19class MeanVar(nn.Module):20 def __init__(self, d):21 super().__init__()22 self.body = nn.Sequential(nn.Linear(d, 64), nn.ReLU())23 self.mu, self.raw_var = nn.Linear(64, 1), nn.Linear(64, 1)24 def forward(self, x):25 h = self.body(x)26 return self.mu(h).squeeze(-1), F.softplus(self.raw_var(h)).squeeze(-1) + 1e-62728# Poisson: the network outputs log-rates29counts = torch.tensor([0., 1., 1., 2., 2., 2., 3., 5.])30log_rate = torch.full_like(counts, math.log(2.0))31print(nn.PoissonNLLLoss()(log_rate, counts)) # 0.6137 = mean(exp(u) - y*u)Use when
- +A principled, general recipe: one loss per noise model, with known statistical properties (consistency, efficiency)
- +Can learn uncertainty (variance, rate, mixture weights), not just a point prediction
- +Gradients on canonical parameters are simply mean minus observation
- +Losses from different models are comparable as log-likelihoods on held-out data
Avoid when
- −Only as good as the assumed distribution; a wrong noise model gives a confident wrong fit
- −Learned variances can collapse or grow to dodge hard examples; needs floors, warm-up or β-NLL
- −PyTorch
NLLLosssilently accepts non-log-probabilities and returns meaningless values - −Unbounded: one very improbable observation can dominate the mean
Check yourself
A regression model outputs (μ, σ²) and is trained with GaussianNLLLoss. On one example its error is |y − μ| = 2. Holding μ fixed, what variance minimizes that example's loss?
Check yourself
Why does nn.NLLLoss(F.softmax(z, dim=1), y) produce negative losses?
Remember
- ✓NLL : choose a distribution, let the network output its parameters; minimizing the mean is maximum likelihood.
- ✓Categorical → cross-entropy; Gaussian with fixed → MSE; Laplace → MAE; Bernoulli → BCE; Poisson → .
- ✓PyTorch
NLLLossis the categorical case on log-probabilities:log_softmax+NLLLoss=CrossEntropyLoss. - ✓
GaussianNLLLosslearns a variance: , optimal squared error; watch for variance-induced gradient shrinkage. - ✓For canonical parameters (logit, logits, mean, log-rate) the gradient is prediction minus target.