KL divergence loss

KL(p‖q) trains a predicted distribution q to match a full target distribution p: soft labels, a teacher's outputs, a prior. In PyTorch the input must be log q, use reduction="batchmean", and scale distillation by T².

11 min read1012 wordscore0 viewsupdated 2026-10-11

Definition

KL divergence loss uses the Kullback–Leibler divergence KL(p ∥ q)=∑kpklog⁡pkqk\mathrm{KL}(p\,\|\,q)=\sum_k p_k\log\frac{p_k}{q_k} as a training objective. Here pp is a target distribution and qq is the model's predicted distribution. It is ≥0\ge0, and it is 0 only when q=pq=p. As a loss it is used when the target is a whole distribution rather than a single class: soft labels, a teacher network's outputs in knowledge distillation, or the prior that a VAE's posterior is pulled toward.

The divergence itself comes from Kullback and Leibler (1951), and the math article Cross-Entropy & KL Divergence covers its properties: non-negativity, asymmetry, and the identity H(p,q)=H(p)+KL(p∥q)H(p,q)=H(p)+\mathrm{KL}(p\|q). This article is about using it as a loss. That means choosing a direction, understanding its gradient, calling nn.KLDivLoss correctly, and the two places deep learning uses it most: knowledge distillation (Hinton, Vinyals & Dean, 2015) and the KL term of the VAE (Kingma & Welling, 2013).

Formula and gradient

LKL=KL(p ∥ qθ)=∑kpklog⁡pk−∑kpklog⁡qθ,k=−H(p)+H(p,qθ),∂LKL∂zk=qk−pk\mathcal L_{\mathrm{KL}}=\mathrm{KL}(p\,\|\,q_\theta)=\sum_k p_k\log p_k-\sum_k p_k\log q_{\theta,k}=-H(p)+H(p,q_\theta),\qquad \frac{\partial\mathcal L_{\mathrm{KL}}}{\partial z_k}=q_k-p_k
pp
target distribution (fixed, or detached from the graph)
qθ=softmax⁡(z)q_\theta=\operatorname{softmax}(z)
model distribution from logits zz
H(p)H(p)
entropy of the target, constant in θ\theta, so it changes the loss value but not the gradient
qk−pkq_k-p_k
the same gradient as softmax cross-entropy with soft labels

Since H(p)H(p) does not depend on the model, minimizing forward KL and minimizing cross-entropy against pp give identical gradients. The difference is the reported value. KL reaches 0 at a perfect fit, while soft-label cross-entropy bottoms out at H(p)>0H(p)>0, which makes KL easier to read when the target is soft. For a one-hot pp, H(p)=0H(p)=0 and the two coincide exactly.

At a glance

Task
match a predicted distribution to a target distribution
Range
[0,∞][0,\infty]; infinite if qk=0q_k=0 where pk>0p_k>0
Symmetric
no: KL(p∥q)≠KL(q∥p)\mathrm{KL}(p\|q)\ne\mathrm{KL}(q\|p) in general
Gradient
q−pq-p w.r.t. logits (forward KL, softmax output)
PyTorch
nn.KLDivLoss(reduction="batchmean")(log_q, p); input in log-space, target as probabilities (or log_target=True)
Units
nats with natural logs (PyTorch); divide by ln⁡2\ln2 for bits

Forward vs reverse KL

The direction matters whenever the model family cannot represent the target exactly. Forward KL, KL(p∥q)\mathrm{KL}(p\|q), takes the expectation under pp. It is infinite wherever q=0q=0 but p>0p>0, so qq is forced to put mass everywhere pp does: it is mass-covering (mean-seeking). Maximum likelihood, soft-label training and distillation all use this direction.

Reverse KL, KL(q∥p)\mathrm{KL}(q\|p), takes the expectation under qq. It only penalizes qq for putting mass where pp has little, so qq may ignore whole modes of pp and lock onto one: it is mode-seeking. Variational inference (and so the VAE's ELBO), the KL penalty to a reference policy in RLHF, and many policy-regularization terms use this direction, mostly because it only needs samples from qq.

a+0.40b+0.18c-0.06d-0.10e-0.10■ p (true)■ q (model)drag bar tops · numbers below = pᵢ log₂(pᵢ/qᵢ)
H(p, q)
2.009 + 0.313 = 2.322
H(q, p)
2.322 + 0.366 = 2.688

■ entropy + ■ KL = cross-entropy (bits). KL(p‖q) = 0.313 vs KL(q‖p) = 0.366 — not symmetric.

fig.The math article's figure. The first view lets you drag p and q and compare KL(p‖q) with KL(q‖p). Switch to the fitting view to fit one Gaussian to a bimodal target: forward KL covers both modes with a wide Gaussian (σ = 2.09), while reverse KL locks onto one. The same asymmetry decides which direction a loss should use.

nn.KLDivLoss: log-space input, batchmean

PyTorch's nn.KLDivLoss computes the pointwise term p (log⁡p−x)p\,(\log p-x), where xx is the input and pp the target. It is only a KL divergence if you pass x=log⁡qx=\log q, e.g. F.log_softmax(logits, dim=-1). Passing probabilities instead silently computes something else, and that value can even be negative.

The reduction is a second trap. reduction="mean" averages over every element, i.e. batch × classes, so it returns KL divided by the number of classes. reduction="batchmean" sums over classes and divides by the batch size, which is the true mean KL per example. The PyTorch documentation warns about exactly this and recommends "batchmean".

Knowledge distillation and the T² factor

Hinton, Vinyals and Dean (2015) train a small student to match a large teacher's softened output distribution. Both logit vectors are divided by a temperature T>1T>1 before the softmax, which makes the small probabilities of wrong classes visible: the "dark knowledge" that a 2 looks more like a 3 than like a 7. The loss is a weighted sum of the usual hard-label cross-entropy at T=1T=1 and the soft KL term:

L=(1−λ) CE(y,softmax⁡(z))+λ T2 KL(softmax⁡(v/T) ∥ softmax⁡(z/T)),∂ KL∂zk=qk−pkT\mathcal L=(1-\lambda)\,\mathrm{CE}\big(y,\operatorname{softmax}(z)\big)+\lambda\,T^2\,\mathrm{KL}\big(\operatorname{softmax}(v/T)\,\big\|\,\operatorname{softmax}(z/T)\big),\qquad \frac{\partial\,\mathrm{KL}}{\partial z_k}=\frac{q_k-p_k}{T}
v,  zv,\;z
teacher and student logits (the teacher is detached)
TT
temperature; Hinton et al. used values up to 20 on MNIST, and 2–5 is common
λ\lambda
weight on the soft term; Hinton et al. put most of the weight on it
T2T^2
compensates for the soft gradients shrinking as 1/T21/T^2

Why T2T^2? The gradient is (qk−pk)/T(q_k-p_k)/T, and the gap qk−pkq_k-p_k itself also shrinks as TT grows. For large TT, qk−pk≈(zk−vk)/(KT)q_k-p_k\approx(z_k-v_k)/(KT) for zero-mean logits over KK classes, so the gradient falls like 1/T21/T^2. Multiplying by T2T^2 keeps the soft term's contribution roughly constant when you change TT, so λ\lambda does not need re-tuning.

p = softmax(v/T), q = softmax(z/T), T = 4.0■ p teacher ■ q studentT²·∂KL(p‖q)/∂zᵢcatv=5 z=3.00.4290.302−0.507dogv=3 z=3.00.2600.3020.168foxv=1 z=0.00.1580.143−0.060owlv=-1 z=0.00.0960.1430.188carv=-3 z=-1.00.0580.1110.212‖∇z‖ of KL(p‖q) vs T━ T²·KL ━ raw KL11e-11e-21e-314710temperature Tforward KL(p‖q) 0.0515 · reverse KL(q‖p) 0.0540 nats · T²·KL(p‖q) 0.8235nn.KLDivLoss(log q, p): batchmean 0.0515 · mean (÷5 classes) 0.0103bug, KLDivLoss(q, p) without the log: −1.6447 (negative: not a divergence)‖∇‖ raw 0.0380 · with T² 0.6076forward KL = cross-entropy H(p, q) − H(p): same gradient as soft-label cross-entropy
loss
fig.A teacher with logits (5, 3, 1, −1, −3) and an editable student starting at (3, 3, 0, 0, −1). At the default T = 4 the softened target is p = (0.429, 0.260, 0.158, 0.096, 0.058). Forward KL is 0.0515 nats and reverse KL 0.0540, and the T²-scaled loss is 0.8235. The right panel plots the gradient norm against T. Without scaling it falls from 0.530 at T = 1 to 0.038 at T = 4 and 0.0062 at T = 10. With T² it stays between 0.53 and 0.62. The readouts show the PyTorch traps: reduction="mean" reports 0.0103 (KL divided by 5 classes), and passing q instead of log q returns −1.6447. Press "10 GD steps" to train the student on either direction.

The KL term of a VAE

A VAE's loss is the negative ELBO: reconstruction error plus KL(qϕ(z∣x) ∥ p(z))\mathrm{KL}(q_\phi(z\mid x)\,\|\,p(z)). This is a reverse-direction KL from the encoder's Gaussian to the prior N(0,I)\mathcal N(0,I). It is computed in closed form, not with nn.KLDivLoss (which is for discrete distributions given as tensors):

KL(N(μ,diag⁡σ2) ∥ N(0,I))=12∑j(μj2+σj2−1−log⁡σj2)\mathrm{KL}\big(\mathcal N(\mu,\operatorname{diag}\sigma^2)\,\|\,\mathcal N(0,I)\big)=\tfrac12\sum_{j}\big(\mu_j^2+\sigma_j^2-1-\log\sigma_j^2\big)
μj,  log⁡σj2\mu_j,\;\log\sigma_j^2
encoder outputs per latent dimension (predict the log-variance for stability)
∑j\textstyle\sum_j
sum over latent dimensions, then average over the batch, like batchmean

Weighting this term is a design choice. β-VAE (Higgins et al., 2017) multiplies it by β>1\beta>1 to encourage disentangled codes. KL annealing (Bowman et al., 2016) ramps its weight up from 0 so a strong decoder does not learn to ignore zz (posterior collapse). See the Variational autoencoder (VAE) article for the full derivation.

Worked example

kpₖqₖpₖ log(pₖ/qₖ)
10.70.50.2355
20.20.3−0.0811
30.10.2−0.0693
sum110.0851 nats
p = (0.7, 0.2, 0.1) target, q = (0.5, 0.3, 0.2) model (checked numerically in Node)
  1. 1Forward KL(p∥q)=0.0851\mathrm{KL}(p\|q)=0.0851 nats. Individual terms can be negative; only the sum is guaranteed ≥0\ge0.
  2. 2Check via entropies: H(p,q)=0.8869H(p,q)=0.8869 and H(p)=0.8018H(p)=0.8018, and 0.8869−0.8018=0.08510.8869-0.8018=0.0851. Soft-label cross-entropy would report 0.88690.8869 with the same gradient.
  3. 3Reverse KL(q∥p)=0.0920\mathrm{KL}(q\|p)=0.0920: a different number, because KL is not symmetric.
  4. 4The bug: nn.KLDivLoss(reduction="sum")(q, p) with probabilities instead of log-probabilities returns ∑plog⁡p−∑p q=−0.8018−0.43=−1.2318\sum p\log p-\sum p\,q=-0.8018-0.43=-1.2318. A negative "divergence" is the telltale sign.
  5. 5VAE term for one latent dimension with μ=0.5, σ=0.8\mu=0.5,\ \sigma=0.8: 12(0.25+0.64−1−log⁡0.64)=0.1681\tfrac12(0.25+0.64-1-\log0.64)=0.1681 nats. At μ=0, σ=1\mu=0,\ \sigma=1 it is exactly 0.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3p = torch.tensor([[0.7, 0.2, 0.1]]) # target probabilities
4q = torch.tensor([[0.5, 0.3, 0.2]]) # model probabilities
5
6kl = nn.KLDivLoss(reduction="batchmean") # input = log q, target = p
7kl(q.log(), p) # 0.0851 = KL(p||q)
8nn.KLDivLoss(reduction="mean")(q.log(), p) # 0.0284 = KL / 3 classes (wrong scale)
9nn.KLDivLoss(reduction="batchmean", log_target=True)(q.log(), p.log()) # 0.0851 too
10
11def distill_loss(student_logits, teacher_logits, y, T=4.0, lam=0.5):
12 soft = F.kl_div(
13 F.log_softmax(student_logits / T, dim=-1),
14 F.softmax(teacher_logits.detach() / T, dim=-1),
15 reduction="batchmean",
16 ) * T * T # keep gradient scale independent of T
17 hard = F.cross_entropy(student_logits, y)
18 return (1 - lam) * hard + lam * soft
19
20zs = torch.tensor([[3.0, 3.0, 0.0, 0.0, -1.0]])
21zt = torch.tensor([[5.0, 3.0, 1.0, -1.0, -3.0]])
22distill_loss(zs, zt, torch.tensor([0])) # 0.5 * 0.7504 + 0.5 * 0.8235 = 0.7870
23
24# VAE: closed-form KL(N(mu, sigma^2) || N(0, 1)), summed over latents, averaged over the batch
25def vae_kl(mu, logvar):
26 return 0.5 * (mu.pow(2) + logvar.exp() - 1 - logvar).sum(dim=1).mean()
27
28vae_kl(torch.tensor([[0.5]]), torch.tensor([[0.64]]).log()) # 0.1681
29
30# Or let torch.distributions do it
31from torch.distributions import Normal, kl_divergence
32kl_divergence(Normal(0.5, 0.8), Normal(0.0, 1.0)) # 0.1681
KLDivLoss used correctly, a distillation loss, and the VAE KL term

Use when

  • +Uses the full target distribution, not just its argmax: soft labels carry similarity information
  • +Same gradient as soft-label cross-entropy, q−pq-p, but reads 0 at a perfect fit
  • +Principled: forward KL is maximum likelihood, reverse KL is the variational-inference objective
  • +Closed form for many distribution pairs (Gaussians, categoricals) via torch.distributions

Avoid when

  • −Asymmetric: the direction is a modelling choice (mass-covering vs mode-seeking)
  • −Infinite when q gives zero mass to the target's support; needs log-space computation
  • −Easy to misuse in PyTorch: log-space input, argument order, "batchmean"
  • −Distillation adds a temperature and a mixing weight, and needs the T2T^2 factor

Check yourself

A distillation run uses F.kl_div(F.log_softmax(zs/T, -1), F.softmax(zt/T, -1), reduction="batchmean") with T = 8 and no T² factor. Compared with T = 1, what happens to the soft term's gradient?

Check yourself

Which call computes KL(p‖q) for model logits z and target probabilities p?

Remember

  • ✓KL(p∥q)=H(p,q)−H(p)\mathrm{KL}(p\|q)=H(p,q)-H(p): same gradient as soft-label cross-entropy (q−pq-p w.r.t. logits), but 0 at a perfect fit.
  • ✓Forward KL(p∥q)\mathrm{KL}(p\|q) is mass-covering (MLE, distillation); reverse KL(q∥p)\mathrm{KL}(q\|p) is mode-seeking (VI, VAE, RLHF penalties).
  • ✓nn.KLDivLoss: input =log⁡q=\log q (log_softmax), target =p=p, reduction="batchmean".
  • ✓Distillation: soften both distributions with temperature TT and multiply the KL by T2T^2.
  • ✓VAE KL to N(0,I)\mathcal N(0,I) is closed-form: 12∑(μ2+σ2−1−log⁡σ2)\tfrac12\sum(\mu^2+\sigma^2-1-\log\sigma^2).
← back to Deep Learning roadmap