Negative log-likelihood loss

Pick a distribution for the target, let the network output its parameters, and minimize −log p(y | x). Categorical gives cross-entropy, Gaussian gives MSE or, with a learned variance, GaussianNLLLoss, and Poisson gives a loss for counts.

12 min read951 wordscore0 viewsupdated 2026-10-11

Definition

Negative log-likelihood (NLL) is the loss −ln⁡pθ(y∣x)-\ln p_\theta(y\mid x): the network outputs the parameters of a probability distribution over targets, and the loss is minus the log-probability (or log-density) that distribution assigns to the observed target. Averaged over a dataset, minimizing it is maximum-likelihood estimation.

NLL is less a single loss than a recipe for making losses. Choose a noise model, write down its log-density, negate it. The Loss functions overview used the recipe to show that MSE and cross-entropy are both NLLs. Here the recipe is the main subject: the general principle, PyTorch's nn.NLLLoss (which is specifically the categorical case and expects log-probabilities), and two NLLs that ordinary point losses cannot express: a Gaussian whose variance the network predicts, and a Poisson for counts.

Maximum likelihood was introduced and named by R. A. Fisher (1922), with earlier use by Gauss, who derived least squares from normally distributed errors in 1809. Under regularity conditions the maximum-likelihood estimator is consistent and asymptotically efficient, which is the classical argument for it. For neural networks, Nix and Weigend (1994) trained a network to output both a mean and a variance by minimizing the Gaussian NLL, the idea behind nn.GaussianNLLLoss.

The principle

θ^=arg⁡max⁡θ∏i=1npθ(yi∣xi)=arg⁡min⁡θ −1n∑i=1nln⁡pθ(yi∣xi)⏟mean NLL\hat\theta=\arg\max_\theta\prod_{i=1}^n p_\theta(y_i\mid x_i)=\arg\min_\theta\ \underbrace{-\frac1n\sum_{i=1}^n\ln p_\theta(y_i\mid x_i)}_{\text{mean NLL}}
pθ(y∣x)p_\theta(y\mid x)
the model: a distribution over yy whose parameters (mean, variance, rate, class probabilities…) are network outputs
ln⁡\ln
turns the product of many small numbers into a sum that does not underflow, without moving the optimum
1n\frac1n
averaging makes the loss independent of dataset size; it is the cross-entropy between the empirical distribution and the model

Every observation contributes −ln⁡p(yi)-\ln p(y_i): zero only if the model puts probability 1 on it, large if the model thought it improbable. A model cannot cheat by assigning high probability everywhere, because a distribution has total mass 1. Raising the density in one place lowers it elsewhere, so NLL rewards putting mass where the data actually are.

target ydistributionnetwork outputs−log p(y) up to constantsPyTorch
binaryBernoulli(p)logit zz, p=σ(z)p=\sigma(z)softplus⁡(z)−yz\operatorname{softplus}(z)-yzBCEWithLogitsLoss
one of K classesCategorical(p)logits, p=softmax⁡(z)p=\operatorname{softmax}(z)−ln⁡py-\ln p_yCrossEntropyLoss, NLLLoss
real, fixed noiseN(μ,σ2)\mathcal N(\mu,\sigma^2), σ\sigma constμ\mu(y−μ)2/2σ2(y-\mu)^2/2\sigma^2MSELoss
real, learned noiseN(μ,σ2)\mathcal N(\mu,\sigma^2)μ\mu, σ2>0\sigma^2>012[ln⁡σ2+(y−μ)2/σ2]\tfrac12[\ln\sigma^2+(y-\mu)^2/\sigma^2]GaussianNLLLoss
real, heavy tailsLaplace(μ,b)(\mu,b), bb constμ\mu∣y−μ∣/b|y-\mu|/bL1Loss
count 0, 1, 2…Poisson(λ)(\lambda)ln⁡λ\ln\lambdaλ−yln⁡λ\lambda-y\ln\lambdaPoissonNLLLoss
Common output distributions and the loss each one yields (constants dropped where noted)

nn.NLLLoss: the categorical case

In PyTorch, nn.NLLLoss is not the general recipe. It is the categorical NLL with the log-probabilities supplied by you: for input ℓ∈RN×K\ell\in\mathbb R^{N\times K} and integer targets it returns −ℓi,yi-\ell_{i,y_i}, averaged. It does not take a log and it does not normalize. It is meant to follow nn.LogSoftmax(dim=1); that pair is exactly nn.CrossEntropyLoss.

NLLLoss(ℓ,y)=−∑iwyi ℓi,yi∑iwyi,CrossEntropyLoss(z,y)=NLLLoss(log_softmax(z),y)\texttt{NLLLoss}(\ell,y)=-\frac{\sum_i w_{y_i}\,\ell_{i,y_i}}{\sum_i w_{y_i}},\qquad \texttt{CrossEntropyLoss}(z,y)=\texttt{NLLLoss}(\texttt{log\_softmax}(z),y)
ℓi,k\ell_{i,k}
log-probability of class kk for example ii (must already be normalized)
wkw_k
optional class weight; all 1 by default, giving a plain mean

Separate log_softmax + NLLLoss is useful when the log-probabilities are needed elsewhere (beam search, sampling, a CTC or CRF layer) or when the model's output is a log-probability by construction, for example a mixture whose log-likelihood is computed with logsumexp.

model N(μ = 2.00, σ = 1.00) · stem = p(yᵢ)0123456−log p(yᵢ): 1.24 0.92 0.96 1.32 1.52 1.90 2.92 5.72mean NLL vs μ024680246MLE 3.00mean NLL 2.0639 · ∂/∂μ −1.000 · ∂/∂σ² −0.645 · MLE μ = 3.00, σ² = 1.29nn.GaussianNLLLoss (drops ½log 2π) = 1.1450 = NLL − 0.9189MSE 2.290; shrink σ below the residuals and the (y−μ)²/σ² term explodes
model
fig.Eight observations and a model distribution. Each orange stem is the density the model gives one observation; the loss is the mean of −log of those heights. Gaussian tab: the starting model N(2, 1) has mean NLL 2.0639 and ∂/∂μ = −1.000. 'Fit by MLE' moves to μ = ȳ = 3.00 and σ² = 1.29 (the 1/n variance), where both gradients vanish and the NLL is 1.5463; nn.GaussianNLLLoss reports 0.6273 because it drops ½log 2π = 0.9189. With 'σ fixed at 1' the NLL is exactly ½·MSE + 0.919 (1.5639 at μ = 3): least squares. Overconfidence is expensive: σ = 0.5 at μ = 3 gives 2.81 (the readout shows 2.79 at the slider step nearest σ = 0.50). Poisson tab: counts 0,1,1,2,2,2,3,5 have MLE λ = 2.00 and NLL 1.6960.

Gaussian NLL with a learned variance

MSE assumes every target has the same noise. Often it does not: some inputs are inherently harder to predict. Let the network output a mean μ(x)\mu(x) and a variance σ2(x)>0\sigma^2(x)>0 (usually through a softplus or an exponential), and minimize the Gaussian NLL:

ℓ=12[ln⁡σ2+(y−μ)2σ2]+12ln⁡2π,∂ℓ∂μ=μ−yσ2,∂ℓ∂σ2=12[1σ2−(y−μ)2σ4]\ell=\tfrac12\Big[\ln\sigma^2+\frac{(y-\mu)^2}{\sigma^2}\Big]+\tfrac12\ln2\pi,\qquad \frac{\partial\ell}{\partial\mu}=\frac{\mu-y}{\sigma^2},\qquad\frac{\partial\ell}{\partial\sigma^2}=\frac12\Big[\frac1{\sigma^2}-\frac{(y-\mu)^2}{\sigma^4}\Big]
ln⁡σ2\ln\sigma^2
penalizes claiming large uncertainty
(y−μ)2/σ2(y-\mu)^2/\sigma^2
penalizes errors, discounted where the model admits uncertainty
σ2∗=(y−μ)2\sigma^{2*}=(y-\mu)^2
per example, the best variance equals the squared error; over a set, the mean squared error

The two terms balance: overclaiming confidence (small σ2\sigma^2) makes the error term explode, and hedging (large σ2\sigma^2) costs ln⁡σ2\ln\sigma^2. The result is a regression model that reports its own error bars. PyTorch's nn.GaussianNLLLoss(full=False, eps=1e-6) computes 12[ln⁡max⁡(σ2,ϵ)+(y−μ)2/max⁡(σ2,ϵ)]\tfrac12[\ln\max(\sigma^2,\epsilon)+(y-\mu)^2/\max(\sigma^2,\epsilon)] and adds 12ln⁡2π\tfrac12\ln2\pi only with full=True. It takes the variance, not the standard deviation.

Poisson NLL for counts

Counts (arrivals per hour, photons per pixel, clicks per day) are non-negative integers whose variance grows with their mean. A Poisson model with rate λ(x)>0\lambda(x)>0 fits that. Writing the network's output as u=ln⁡λu=\ln\lambda keeps the rate positive:

−ln⁡Poisson(y∣λ)=λ−yln⁡λ+ln⁡y!=eu−yu+ln⁡y!,∂ℓ∂u=eu−y=λ−y-\ln\mathrm{Poisson}(y\mid\lambda)=\lambda-y\ln\lambda+\ln y!=e^u-yu+\ln y!,\qquad\frac{\partial\ell}{\partial u}=e^u-y=\lambda-y
u=ln⁡λu=\ln\lambda
log-rate, the network output (PyTorch log_input=True, the default)
ln⁡y!\ln y!
constant in the parameters; full=True adds a Stirling approximation of it for y>1y>1
λ−y\lambda-y
the familiar prediction-minus-target gradient, now in log-rate space

The gradient λ−y\lambda-y on the natural parameter is not a coincidence. Bernoulli with a logit, categorical with logits, Gaussian with a mean and Poisson with a log-rate are all exponential-family distributions in their canonical parameter, and for every one of them the NLL gradient is mean minus observation. This is the same structure that generalized linear models use.

Worked example

modelμσ²mean NLL∂/∂μ∂/∂σ²
start2.001.002.0639−1.000−0.645
right mean, σ fixed3.001.001.56390−0.145
MLE3.001.291.546300
overconfident3.000.252.80580−8.320
underconfident3.004.001.773300.085
Eight observations y = (1.2, 2.0, 2.3, 2.9, 3.1, 3.4, 4.0, 5.1); mean NLL in nats (checked numerically in Node)
  1. 1yˉ=3.0\bar y=3.0 and 1n∑(yi−yˉ)2=1.29\frac1n\sum(y_i-\bar y)^2=1.29: the MLE of the mean is the sample mean and of the variance the biased (divide-by-nn) sample variance.
  2. 2At σ2=1\sigma^2=1 the NLL is 12MSE+12ln⁡2π=12(1.29)+0.9189=1.5639\tfrac12\mathrm{MSE}+\tfrac12\ln2\pi=\tfrac12(1.29)+0.9189=1.5639: with a fixed variance, minimizing NLL is minimizing MSE.
  3. 3Freeing σ2\sigma^2 lowers the NLL to 1.5463. A negative ∂/∂σ2\partial/\partial\sigma^2 means "increase the variance"; it is zero exactly at σ2=MSE\sigma^2=\mathrm{MSE}.
  4. 4GaussianNLLLoss at the MLE returns 1.5463−0.9189=0.62731.5463-0.9189=0.6273. For the counts (0,1,1,2,2,2,3,5)(0,1,1,2,2,2,3,5) the Poisson MLE is λ=yˉ=2\lambda=\bar y=2, with mean NLL 1.6960 and PoissonNLLLoss value 0.6137 (no ln⁡y!\ln y! term).
012345678910bar height = N(xᵢ | μ, σ²) · likelihood = product of barslog-likelihood vs μ (σ = 2.40)μ̂ = x̄ = 6.18
log L = -27.17 (max -16.99 at μ̂ = 6.18, σ̂ = 1.32)
fig.From the statistics chapter: the likelihood as a product of per-point densities. Drag μ and σ, or press 'Fit by MLE', and watch the log-likelihood peak at the sample mean and the 1/n standard deviation; the NLL loss is its negative.
python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3# categorical: NLLLoss expects log-probabilities
4z = torch.tensor([[2.0, 0.5, -1.0]])
5y = torch.tensor([1])
6print(nn.NLLLoss()(F.log_softmax(z, dim=1), y)) # 1.7413
7print(nn.CrossEntropyLoss()(z, y)) # 1.7413, same thing fused
8print(nn.NLLLoss()(F.softmax(z, dim=1), y)) # -0.1753 <- wrong input, wrong loss
9
10# Gaussian with a predicted variance (heteroscedastic regression)
11target = torch.tensor([1.2, 2.0, 2.3, 2.9, 3.1, 3.4, 4.0, 5.1])
12mean = torch.full_like(target, 3.0)
13var = torch.full_like(target, 1.29) # variance, not std; must be > 0
14gnll = nn.GaussianNLLLoss() # full=False, eps=1e-6
15print(gnll(mean, target, var)) # 0.6273
16print(nn.GaussianNLLLoss(full=True)(mean, target, var)) # 1.5463 = 0.6273 + 0.5*log(2*pi)
17
18# a two-headed model: softplus keeps the variance positive
19class MeanVar(nn.Module):
20 def __init__(self, d):
21 super().__init__()
22 self.body = nn.Sequential(nn.Linear(d, 64), nn.ReLU())
23 self.mu, self.raw_var = nn.Linear(64, 1), nn.Linear(64, 1)
24 def forward(self, x):
25 h = self.body(x)
26 return self.mu(h).squeeze(-1), F.softplus(self.raw_var(h)).squeeze(-1) + 1e-6
27
28# Poisson: the network outputs log-rates
29counts = torch.tensor([0., 1., 1., 2., 2., 2., 3., 5.])
30log_rate = torch.full_like(counts, math.log(2.0))
31print(nn.PoissonNLLLoss()(log_rate, counts)) # 0.6137 = mean(exp(u) - y*u)
Categorical, Gaussian and Poisson NLL in PyTorch

Use when

  • +A principled, general recipe: one loss per noise model, with known statistical properties (consistency, efficiency)
  • +Can learn uncertainty (variance, rate, mixture weights), not just a point prediction
  • +Gradients on canonical parameters are simply mean minus observation
  • +Losses from different models are comparable as log-likelihoods on held-out data

Avoid when

  • −Only as good as the assumed distribution; a wrong noise model gives a confident wrong fit
  • −Learned variances can collapse or grow to dodge hard examples; needs floors, warm-up or β-NLL
  • −PyTorch NLLLoss silently accepts non-log-probabilities and returns meaningless values
  • −Unbounded: one very improbable observation can dominate the mean

Check yourself

A regression model outputs (μ, σ²) and is trained with GaussianNLLLoss. On one example its error is |y − μ| = 2. Holding μ fixed, what variance minimizes that example's loss?

Check yourself

Why does nn.NLLLoss(F.softmax(z, dim=1), y) produce negative losses?

Remember

  • ✓NLL =−ln⁡pθ(y∣x)=-\ln p_\theta(y\mid x): choose a distribution, let the network output its parameters; minimizing the mean is maximum likelihood.
  • ✓Categorical → cross-entropy; Gaussian with fixed σ\sigma → MSE; Laplace → MAE; Bernoulli → BCE; Poisson → λ−yln⁡λ\lambda-y\ln\lambda.
  • ✓PyTorch NLLLoss is the categorical case on log-probabilities: log_softmax + NLLLoss = CrossEntropyLoss.
  • ✓GaussianNLLLoss learns a variance: 12[ln⁡σ2+(y−μ)2/σ2]\tfrac12[\ln\sigma^2+(y-\mu)^2/\sigma^2], optimal σ2=\sigma^2= squared error; watch for variance-induced gradient shrinkage.
  • ✓For canonical parameters (logit, logits, mean, log-rate) the gradient is prediction minus target.
← back to Deep Learning roadmap