Definition
KL divergence loss uses the Kullback–Leibler divergence as a training objective. Here is a target distribution and is the model's predicted distribution. It is , and it is 0 only when . As a loss it is used when the target is a whole distribution rather than a single class: soft labels, a teacher network's outputs in knowledge distillation, or the prior that a VAE's posterior is pulled toward.
The divergence itself comes from Kullback and Leibler (1951), and the math article Cross-Entropy & KL Divergence covers its properties: non-negativity, asymmetry, and the identity . This article is about using it as a loss. That means choosing a direction, understanding its gradient, calling nn.KLDivLoss correctly, and the two places deep learning uses it most: knowledge distillation (Hinton, Vinyals & Dean, 2015) and the KL term of the VAE (Kingma & Welling, 2013).
Formula and gradient
- target distribution (fixed, or detached from the graph)
- model distribution from logits
- entropy of the target, constant in , so it changes the loss value but not the gradient
- the same gradient as softmax cross-entropy with soft labels
Since does not depend on the model, minimizing forward KL and minimizing cross-entropy against give identical gradients. The difference is the reported value. KL reaches 0 at a perfect fit, while soft-label cross-entropy bottoms out at , which makes KL easier to read when the target is soft. For a one-hot , and the two coincide exactly.
At a glance
- Task
- match a predicted distribution to a target distribution
- Range
- ; infinite if where
- Symmetric
- no: in general
- Gradient
- w.r.t. logits (forward KL, softmax output)
- PyTorch
nn.KLDivLoss(reduction="batchmean")(log_q, p); input in log-space, target as probabilities (orlog_target=True)- Units
- nats with natural logs (PyTorch); divide by for bits
Forward vs reverse KL
The direction matters whenever the model family cannot represent the target exactly. Forward KL, , takes the expectation under . It is infinite wherever but , so is forced to put mass everywhere does: it is mass-covering (mean-seeking). Maximum likelihood, soft-label training and distillation all use this direction.
Reverse KL, , takes the expectation under . It only penalizes for putting mass where has little, so may ignore whole modes of and lock onto one: it is mode-seeking. Variational inference (and so the VAE's ELBO), the KL penalty to a reference policy in RLHF, and many policy-regularization terms use this direction, mostly because it only needs samples from .
■ entropy + ■ KL = cross-entropy (bits). KL(p‖q) = 0.313 vs KL(q‖p) = 0.366 — not symmetric.
nn.KLDivLoss: log-space input, batchmean
PyTorch's nn.KLDivLoss computes the pointwise term , where is the input and the target. It is only a KL divergence if you pass , e.g. F.log_softmax(logits, dim=-1). Passing probabilities instead silently computes something else, and that value can even be negative.
The reduction is a second trap. reduction="mean" averages over every element, i.e. batch × classes, so it returns KL divided by the number of classes. reduction="batchmean" sums over classes and divides by the batch size, which is the true mean KL per example. The PyTorch documentation warns about exactly this and recommends "batchmean".
Knowledge distillation and the T² factor
Hinton, Vinyals and Dean (2015) train a small student to match a large teacher's softened output distribution. Both logit vectors are divided by a temperature before the softmax, which makes the small probabilities of wrong classes visible: the "dark knowledge" that a 2 looks more like a 3 than like a 7. The loss is a weighted sum of the usual hard-label cross-entropy at and the soft KL term:
- teacher and student logits (the teacher is detached)
- temperature; Hinton et al. used values up to 20 on MNIST, and 2–5 is common
- weight on the soft term; Hinton et al. put most of the weight on it
- compensates for the soft gradients shrinking as
Why ? The gradient is , and the gap itself also shrinks as grows. For large , for zero-mean logits over classes, so the gradient falls like . Multiplying by keeps the soft term's contribution roughly constant when you change , so does not need re-tuning.
The KL term of a VAE
A VAE's loss is the negative ELBO: reconstruction error plus . This is a reverse-direction KL from the encoder's Gaussian to the prior . It is computed in closed form, not with nn.KLDivLoss (which is for discrete distributions given as tensors):
- encoder outputs per latent dimension (predict the log-variance for stability)
- sum over latent dimensions, then average over the batch, like batchmean
Weighting this term is a design choice. β-VAE (Higgins et al., 2017) multiplies it by to encourage disentangled codes. KL annealing (Bowman et al., 2016) ramps its weight up from 0 so a strong decoder does not learn to ignore (posterior collapse). See the Variational autoencoder (VAE) article for the full derivation.
Worked example
| k | pₖ | qₖ | pₖ log(pₖ/qₖ) |
|---|---|---|---|
| 1 | 0.7 | 0.5 | 0.2355 |
| 2 | 0.2 | 0.3 | −0.0811 |
| 3 | 0.1 | 0.2 | −0.0693 |
| sum | 1 | 1 | 0.0851 nats |
- 1Forward nats. Individual terms can be negative; only the sum is guaranteed .
- 2Check via entropies: and , and . Soft-label cross-entropy would report with the same gradient.
- 3Reverse : a different number, because KL is not symmetric.
- 4The bug:
nn.KLDivLoss(reduction="sum")(q, p)with probabilities instead of log-probabilities returns . A negative "divergence" is the telltale sign. - 5VAE term for one latent dimension with : nats. At it is exactly 0.
1import torch, torch.nn as nn, torch.nn.functional as F23p = torch.tensor([[0.7, 0.2, 0.1]]) # target probabilities4q = torch.tensor([[0.5, 0.3, 0.2]]) # model probabilities56kl = nn.KLDivLoss(reduction="batchmean") # input = log q, target = p7kl(q.log(), p) # 0.0851 = KL(p||q)8nn.KLDivLoss(reduction="mean")(q.log(), p) # 0.0284 = KL / 3 classes (wrong scale)9nn.KLDivLoss(reduction="batchmean", log_target=True)(q.log(), p.log()) # 0.0851 too1011def distill_loss(student_logits, teacher_logits, y, T=4.0, lam=0.5):12 soft = F.kl_div(13 F.log_softmax(student_logits / T, dim=-1),14 F.softmax(teacher_logits.detach() / T, dim=-1),15 reduction="batchmean",16 ) * T * T # keep gradient scale independent of T17 hard = F.cross_entropy(student_logits, y)18 return (1 - lam) * hard + lam * soft1920zs = torch.tensor([[3.0, 3.0, 0.0, 0.0, -1.0]])21zt = torch.tensor([[5.0, 3.0, 1.0, -1.0, -3.0]])22distill_loss(zs, zt, torch.tensor([0])) # 0.5 * 0.7504 + 0.5 * 0.8235 = 0.78702324# VAE: closed-form KL(N(mu, sigma^2) || N(0, 1)), summed over latents, averaged over the batch25def vae_kl(mu, logvar):26 return 0.5 * (mu.pow(2) + logvar.exp() - 1 - logvar).sum(dim=1).mean()2728vae_kl(torch.tensor([[0.5]]), torch.tensor([[0.64]]).log()) # 0.16812930# Or let torch.distributions do it31from torch.distributions import Normal, kl_divergence32kl_divergence(Normal(0.5, 0.8), Normal(0.0, 1.0)) # 0.1681Use when
- +Uses the full target distribution, not just its argmax: soft labels carry similarity information
- +Same gradient as soft-label cross-entropy, , but reads 0 at a perfect fit
- +Principled: forward KL is maximum likelihood, reverse KL is the variational-inference objective
- +Closed form for many distribution pairs (Gaussians, categoricals) via torch.distributions
Avoid when
- −Asymmetric: the direction is a modelling choice (mass-covering vs mode-seeking)
- −Infinite when q gives zero mass to the target's support; needs log-space computation
- −Easy to misuse in PyTorch: log-space input, argument order,
"batchmean" - −Distillation adds a temperature and a mixing weight, and needs the factor
Check yourself
A distillation run uses F.kl_div(F.log_softmax(zs/T, -1), F.softmax(zt/T, -1), reduction="batchmean") with T = 8 and no T² factor. Compared with T = 1, what happens to the soft term's gradient?
Check yourself
Which call computes KL(p‖q) for model logits z and target probabilities p?
Remember
- ✓: same gradient as soft-label cross-entropy ( w.r.t. logits), but 0 at a perfect fit.
- ✓Forward is mass-covering (MLE, distillation); reverse is mode-seeking (VI, VAE, RLHF penalties).
- ✓
nn.KLDivLoss: input (log_softmax), target ,reduction="batchmean". - ✓Distillation: soften both distributions with temperature and multiply the KL by .
- ✓VAE KL to is closed-form: .