# KL divergence loss

> KL(p‖q) trains a predicted distribution q to match a full target distribution p: soft labels, a teacher's outputs, a prior. In PyTorch the input must be log q, use reduction="batchmean", and scale distillation by T².

Source: https://www.mlfoundry.in/learn/kl-divergence-loss  
Section: Deep Learning › Loss Functions  
Level: core · 11 min read · updated 2026-10-11

**KL divergence loss:** uses the Kullback–Leibler divergence $\mathrm{KL}(p\,\|\,q)=\sum_k p_k\log\frac{p_k}{q_k}$ as a training objective. Here $p$ is a target distribution and $q$ is the model's predicted distribution. It is $\ge0$, and it is 0 only when $q=p$. As a loss it is used when the target is a **whole distribution** rather than a single class: soft labels, a teacher network's outputs in knowledge distillation, or the prior that a VAE's posterior is pulled toward.

The divergence itself comes from Kullback and Leibler (1951), and the math article **Cross-Entropy & KL Divergence** covers its properties: non-negativity, asymmetry, and the identity $H(p,q)=H(p)+\mathrm{KL}(p\|q)$. This article is about using it as a **loss**. That means choosing a direction, understanding its gradient, calling `nn.KLDivLoss` correctly, and the two places deep learning uses it most: knowledge distillation (Hinton, Vinyals & Dean, 2015) and the KL term of the VAE (Kingma & Welling, 2013).

## Formula and gradient

$$
\mathcal L_{\mathrm{KL}}=\mathrm{KL}(p\,\|\,q_\theta)=\sum_k p_k\log p_k-\sum_k p_k\log q_{\theta,k}=-H(p)+H(p,q_\theta),\qquad \frac{\partial\mathcal L_{\mathrm{KL}}}{\partial z_k}=q_k-p_k
$$

- $p$: target distribution (fixed, or detached from the graph)
- $q_\theta=\operatorname{softmax}(z)$: model distribution from logits $z$
- $H(p)$: entropy of the target, constant in $\theta$, so it changes the loss value but not the gradient
- $q_k-p_k$: the same gradient as softmax cross-entropy with soft labels

Since $H(p)$ does not depend on the model, minimizing forward KL and minimizing cross-entropy against $p$ give **identical gradients**. The difference is the reported value. KL reaches 0 at a perfect fit, while soft-label cross-entropy bottoms out at $H(p)>0$, which makes KL easier to read when the target is soft. For a one-hot $p$, $H(p)=0$ and the two coincide exactly.

**At a glance**

- **Task:** match a predicted distribution to a target distribution
- **Range:** $[0,\infty]$; infinite if $q_k=0$ where $p_k>0$
- **Symmetric:** no: $\mathrm{KL}(p\|q)\ne\mathrm{KL}(q\|p)$ in general
- **Gradient:** $q-p$ w.r.t. logits (forward KL, softmax output)
- **PyTorch:** `nn.KLDivLoss(reduction="batchmean")(log_q, p)`; input in log-space, target as probabilities (or `log_target=True`)
- **Units:** nats with natural logs (PyTorch); divide by $\ln2$ for bits

## Forward vs reverse KL

The direction matters whenever the model family **cannot** represent the target exactly. **Forward KL**, $\mathrm{KL}(p\|q)$, takes the expectation under $p$. It is infinite wherever $q=0$ but $p>0$, so $q$ is forced to put mass everywhere $p$ does: it is **mass-covering** (mean-seeking). Maximum likelihood, soft-label training and distillation all use this direction.

**Reverse KL**, $\mathrm{KL}(q\|p)$, takes the expectation under $q$. It only penalizes $q$ for putting mass where $p$ has little, so $q$ may ignore whole modes of $p$ and lock onto one: it is **mode-seeking**. Variational inference (and so the VAE's ELBO), the KL penalty to a reference policy in RLHF, and many policy-regularization terms use this direction, mostly because it only needs samples from $q$.

> Figure: The math article's figure. The first view lets you drag p and q and compare KL(p‖q) with KL(q‖p). Switch to the fitting view to fit one Gaussian to a bimodal target: forward KL covers both modes with a wide Gaussian (σ = 2.09), while reverse KL locks onto one. The same asymmetry decides which direction a loss should use.

> **With a flexible model the direction hardly matters:** If the model can represent $p$ exactly (for example, free logits for a softmax over a finite set of classes), both directions have the same minimizer $q=p$. In the figure below, ten gradient steps on either direction take the student from KL ≈ 0.05 to about 0.002. The direction becomes a modelling decision only when $q$ is restricted, such as a single Gaussian or a small student network.

## nn.KLDivLoss: log-space input, batchmean

PyTorch's `nn.KLDivLoss` computes the pointwise term $p\,(\log p-x)$, where $x$ is the **input** and $p$ the **target**. It is only a KL divergence if you pass $x=\log q$, e.g. `F.log_softmax(logits, dim=-1)`. Passing probabilities instead silently computes something else, and that value can even be negative.

The reduction is a second trap. `reduction="mean"` averages over **every element**, i.e. batch × classes, so it returns KL divided by the number of classes. `reduction="batchmean"` sums over classes and divides by the batch size, which is the true mean KL per example. The PyTorch documentation warns about exactly this and recommends `"batchmean"`.

## Knowledge distillation and the T² factor

Hinton, Vinyals and Dean (2015) train a small **student** to match a large **teacher**'s softened output distribution. Both logit vectors are divided by a temperature $T>1$ before the softmax, which makes the small probabilities of wrong classes visible: the "dark knowledge" that a 2 looks more like a 3 than like a 7. The loss is a weighted sum of the usual hard-label cross-entropy at $T=1$ and the soft KL term:

$$
\mathcal L=(1-\lambda)\,\mathrm{CE}\big(y,\operatorname{softmax}(z)\big)+\lambda\,T^2\,\mathrm{KL}\big(\operatorname{softmax}(v/T)\,\big\|\,\operatorname{softmax}(z/T)\big),\qquad \frac{\partial\,\mathrm{KL}}{\partial z_k}=\frac{q_k-p_k}{T}
$$

- $v,\;z$: teacher and student logits (the teacher is detached)
- $T$: temperature; Hinton et al. used values up to 20 on MNIST, and 2–5 is common
- $\lambda$: weight on the soft term; Hinton et al. put most of the weight on it
- $T^2$: compensates for the soft gradients shrinking as $1/T^2$

Why $T^2$? The gradient is $(q_k-p_k)/T$, and the gap $q_k-p_k$ itself also shrinks as $T$ grows. For large $T$, $q_k-p_k\approx(z_k-v_k)/(KT)$ for zero-mean logits over $K$ classes, so the gradient falls like $1/T^2$. Multiplying by $T^2$ keeps the soft term's contribution roughly constant when you change $T$, so $\lambda$ does not need re-tuning.

> Figure: A teacher with logits (5, 3, 1, −1, −3) and an editable student starting at (3, 3, 0, 0, −1). At the default T = 4 the softened target is p = (0.429, 0.260, 0.158, 0.096, 0.058). Forward KL is 0.0515 nats and reverse KL 0.0540, and the T²-scaled loss is 0.8235. The right panel plots the gradient norm against T. Without scaling it falls from 0.530 at T = 1 to 0.038 at T = 4 and 0.0062 at T = 10. With T² it stays between 0.53 and 0.62. The readouts show the PyTorch traps: reduction="mean" reports 0.0103 (KL divided by 5 classes), and passing q instead of log q returns −1.6447. Press "10 GD steps" to train the student on either direction.

## The KL term of a VAE

A VAE's loss is the negative ELBO: reconstruction error plus $\mathrm{KL}(q_\phi(z\mid x)\,\|\,p(z))$. This is a **reverse-direction** KL from the encoder's Gaussian to the prior $\mathcal N(0,I)$. It is computed in closed form, not with `nn.KLDivLoss` (which is for discrete distributions given as tensors):

$$
\mathrm{KL}\big(\mathcal N(\mu,\operatorname{diag}\sigma^2)\,\|\,\mathcal N(0,I)\big)=\tfrac12\sum_{j}\big(\mu_j^2+\sigma_j^2-1-\log\sigma_j^2\big)
$$

- $\mu_j,\;\log\sigma_j^2$: encoder outputs per latent dimension (predict the log-variance for stability)
- $\textstyle\sum_j$: sum over latent dimensions, then average over the batch, like batchmean

Weighting this term is a design choice. β-VAE (Higgins et al., 2017) multiplies it by $\beta>1$ to encourage disentangled codes. KL annealing (Bowman et al., 2016) ramps its weight up from 0 so a strong decoder does not learn to ignore $z$ (posterior collapse). See the **Variational autoencoder (VAE)** article for the full derivation.

## Worked example

p = (0.7, 0.2, 0.1) target, q = (0.5, 0.3, 0.2) model (checked numerically in Node)

| k | pₖ | qₖ | pₖ log(pₖ/qₖ) |
| --- | --- | --- | --- |
| 1 | 0.7 | 0.5 | 0.2355 |
| 2 | 0.2 | 0.3 | −0.0811 |
| 3 | 0.1 | 0.2 | −0.0693 |
| sum | 1 | 1 | **0.0851** nats |

1. Forward $\mathrm{KL}(p\|q)=0.0851$ nats. Individual terms can be negative; only the sum is guaranteed $\ge0$.
2. Check via entropies: $H(p,q)=0.8869$ and $H(p)=0.8018$, and $0.8869-0.8018=0.0851$. Soft-label cross-entropy would report $0.8869$ with the same gradient.
3. Reverse $\mathrm{KL}(q\|p)=0.0920$: a different number, because KL is not symmetric.
4. The bug: `nn.KLDivLoss(reduction="sum")(q, p)` with probabilities instead of log-probabilities returns $\sum p\log p-\sum p\,q=-0.8018-0.43=-1.2318$. A negative "divergence" is the telltale sign.
5. VAE term for one latent dimension with $\mu=0.5,\ \sigma=0.8$: $\tfrac12(0.25+0.64-1-\log0.64)=0.1681$ nats. At $\mu=0,\ \sigma=1$ it is exactly 0.

```python
import torch, torch.nn as nn, torch.nn.functional as F

p = torch.tensor([[0.7, 0.2, 0.1]])                 # target probabilities
q = torch.tensor([[0.5, 0.3, 0.2]])                 # model probabilities

kl = nn.KLDivLoss(reduction="batchmean")            # input = log q, target = p
kl(q.log(), p)                                      # 0.0851  = KL(p||q)
nn.KLDivLoss(reduction="mean")(q.log(), p)          # 0.0284  = KL / 3 classes  (wrong scale)
nn.KLDivLoss(reduction="batchmean", log_target=True)(q.log(), p.log())   # 0.0851 too

def distill_loss(student_logits, teacher_logits, y, T=4.0, lam=0.5):
    soft = F.kl_div(
        F.log_softmax(student_logits / T, dim=-1),
        F.softmax(teacher_logits.detach() / T, dim=-1),
        reduction="batchmean",
    ) * T * T                                       # keep gradient scale independent of T
    hard = F.cross_entropy(student_logits, y)
    return (1 - lam) * hard + lam * soft

zs = torch.tensor([[3.0, 3.0, 0.0, 0.0, -1.0]])
zt = torch.tensor([[5.0, 3.0, 1.0, -1.0, -3.0]])
distill_loss(zs, zt, torch.tensor([0]))            # 0.5 * 0.7504 + 0.5 * 0.8235 = 0.7870

# VAE: closed-form KL(N(mu, sigma^2) || N(0, 1)), summed over latents, averaged over the batch
def vae_kl(mu, logvar):
    return 0.5 * (mu.pow(2) + logvar.exp() - 1 - logvar).sum(dim=1).mean()

vae_kl(torch.tensor([[0.5]]), torch.tensor([[0.64]]).log())   # 0.1681

# Or let torch.distributions do it
from torch.distributions import Normal, kl_divergence
kl_divergence(Normal(0.5, 0.8), Normal(0.0, 1.0))  # 0.1681
```

KLDivLoss used correctly, a distillation loss, and the VAE KL term

> **Input in log-space, target in probability space:** `nn.KLDivLoss(input, target)` expects **log-probabilities** as input and, by default, **probabilities** as target. The argument order is the reverse of the maths: `kl(log_q, p)` computes $\mathrm{KL}(p\|q)$. Pass `log_target=True` if your target is also in log-space, which is more stable when the target has tiny probabilities.

> **reduction="mean" is not the mean KL:** With $K$ classes, `"mean"` divides the summed KL by batch × $K$, so the loss and its gradient are $K$ times too small. For 1000 ImageNet classes that is a 1000× smaller effective learning rate on the distillation term. Use `"batchmean"`.

> **Zeros in q:** Forward KL is infinite when $q_k=0$ and $p_k>0$. Always compute $\log q$ with `log_softmax` on logits, never as `softmax(...).log()`, which underflows to $-\infty$ for very negative logits and turns the loss into `inf` or `nan`.

> **Where else it appears:** Label smoothing is cross-entropy against a smoothed target, i.e. forward KL plus a constant. RLHF methods such as PPO-based fine-tuning penalize $\mathrm{KL}(\pi_\theta\|\pi_{\text{ref}})$ to keep the policy near the reference model. Self-distillation and semi-supervised consistency losses match a model's predictions on two views of an input with a KL term.

**Strengths**

- Uses the full target distribution, not just its argmax: soft labels carry similarity information
- Same gradient as soft-label cross-entropy, $q-p$, but reads 0 at a perfect fit
- Principled: forward KL is maximum likelihood, reverse KL is the variational-inference objective
- Closed form for many distribution pairs (Gaussians, categoricals) via torch.distributions

**Limitations**

- Asymmetric: the direction is a modelling choice (mass-covering vs mode-seeking)
- Infinite when q gives zero mass to the target's support; needs log-space computation
- Easy to misuse in PyTorch: log-space input, argument order, `"batchmean"`
- Distillation adds a temperature and a mixing weight, and needs the $T^2$ factor

**Question:** A distillation run uses `F.kl_div(F.log_softmax(zs/T, -1), F.softmax(zt/T, -1), reduction="batchmean")` with T = 8 and no T² factor. Compared with T = 1, what happens to the soft term's gradient?

- A. It becomes much smaller, roughly by a factor of T² = 64 at large T
- B. It becomes 8 times larger
- C. Nothing: temperature only changes the loss value
- D. It changes sign

**Answer:** A. It becomes much smaller, roughly by a factor of T² = 64 at large T — The gradient is $(q-p)/T$, and for large $T$ the gap $q-p$ itself shrinks like $1/T$, so the total falls like $1/T^2$. In the figure the norm drops from 0.530 at T = 1 to 0.0062 at T = 10. Multiplying the loss by $T^2$ restores the scale.

**Question:** Which call computes KL(p‖q) for model logits `z` and target probabilities `p`?

- A. nn.KLDivLoss(reduction="batchmean")(F.log_softmax(z, -1), p)
- B. nn.KLDivLoss(reduction="batchmean")(F.softmax(z, -1), p)
- C. nn.KLDivLoss(reduction="batchmean")(p.log(), F.softmax(z, -1))
- D. nn.KLDivLoss(reduction="mean")(F.log_softmax(z, -1), p)

**Answer:** A. nn.KLDivLoss(reduction="batchmean")(F.log_softmax(z, -1), p) — The input must be $\log q$ and the target $p$. Option 2 forgets the log. Option 3 swaps the roles and computes the reverse direction, $\mathrm{KL}(q\|p)$. Option 4 divides by the number of classes.

**Key points**

- $\mathrm{KL}(p\|q)=H(p,q)-H(p)$: same gradient as soft-label cross-entropy ($q-p$ w.r.t. logits), but 0 at a perfect fit.
- Forward $\mathrm{KL}(p\|q)$ is mass-covering (MLE, distillation); reverse $\mathrm{KL}(q\|p)$ is mode-seeking (VI, VAE, RLHF penalties).
- `nn.KLDivLoss`: input $=\log q$ (`log_softmax`), target $=p$, `reduction="batchmean"`.
- Distillation: soften both distributions with temperature $T$ and multiply the KL by $T^2$.
- VAE KL to $\mathcal N(0,I)$ is closed-form: $\tfrac12\sum(\mu^2+\sigma^2-1-\log\sigma^2)$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
