# Cross-Entropy & KL Divergence

> H(p, q) = H(p) + KL(p‖q), so minimizing cross-entropy over q minimizes the KL divergence to p.

Source: https://www.mlfoundry.in/learn/cross-entropy-kl-divergence  
Section: Mathematics › Information Theory  
Level: core · 6 min read · updated 2026-10-03

**Cross-entropy:** $H(p,q)=-\sum_x p(x)\log q(x)$ is the average cost of encoding outcomes drawn from $p$ with a code designed for $q$. The **Kullback–Leibler divergence** $\mathrm{KL}(p\,\|\,q)=H(p,q)-H(p)$ is the extra cost of using the wrong distribution: zero when $q=p$, positive otherwise.

- **Notation:** $H(p,q)$; $D_{\mathrm{KL}}(p\,\|\,q)$ — read "KL of $p$ from $q$"
- **Units:** bits with log₂, nats with ln; deep-learning libraries use nats
- **Key property:** $\mathrm{KL}(p\|q)\ge0$, with equality iff $p=q$ (Gibbs' inequality)
- **Not a distance:** $\mathrm{KL}(p\|q)\ne\mathrm{KL}(q\|p)$ in general; no triangle inequality
- **Prerequisites:** entropy, expectation, logarithms
- **Used in:** classification loss, language-model perplexity, VAEs, distillation, RLHF

## The intuition

Outcomes come from the true distribution $p$, but your model believes $q$. Each outcome $x$ costs $-\log q(x)$ bits under your model — a lot when the model thought $x$ was unlikely. Averaged over the real outcomes, you pay $H(p,q)$. The best anyone could pay is $H(p)$, reached only by the true distribution. The gap, KL, is the price of being wrong.

## The math

$$
H(p,q)=\mathbb{E}_{x\sim p}\big[-\log q(x)\big],\qquad \mathrm{KL}(p\,\|\,q)=\sum_x p(x)\log\frac{p(x)}{q(x)}=H(p,q)-H(p)
$$

- $p$: true (data) distribution
- $q$: model distribution

**Gibbs' inequality.** Over the outcomes where $p>0$, Jensen's inequality for the concave $\log$ gives $-\mathrm{KL}(p\|q)=\sum p\log\frac{q}{p}\le\log\sum p\,\frac{q}{p}=\log\sum q\le\log1=0$. So $\mathrm{KL}\ge0$, with equality only when $q=p$.

**Why it is the classification loss.** For a labelled example, $p$ is one-hot on the true class $y$, so $H(p,q)=-\log q_y$: the negative log-likelihood of the correct class. Averaged over the data, minimizing cross-entropy *is* maximum likelihood. And because $H(p)$ does not depend on the model, minimizing cross-entropy and minimizing KL to the data are the same optimization. With $q=\mathrm{softmax}(z)$, the gradient with respect to the logits is simply:

$$
\frac{\partial H(p,q)}{\partial z}=q-p
$$

**Asymmetry.** $\mathrm{KL}(p\|q)$ averages over $p$, so it is infinite wherever $p>0$ but $q=0$: the model must cover everything the data does (*mode-covering*). $\mathrm{KL}(q\|p)$ averages over $q$, so it punishes putting mass where $p\approx0$ and ignores modes $q$ skips (*mode-seeking*).

**Gaussians** have a closed form, in nats — the KL term of every VAE uses it with $\mathcal N(0,1)$ as the prior:

$$
\mathrm{KL}\big(\mathcal N(\mu_1,\sigma_1^2)\,\|\,\mathcal N(\mu_2,\sigma_2^2)\big)=\log\frac{\sigma_2}{\sigma_1}+\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2}-\frac12
$$

## See it

> Figure: Tab 1: drag p and q. Bars show cross-entropy = entropy + KL in both directions; "q misses e" makes KL(p‖q) infinite. Tab 2: fitting one Gaussian to a bimodal p — forward KL covers both modes (σ = 2.09, KL(p‖q) = 0.80 bits), reverse KL locks onto one (KL(q‖p) = 1.00 bit).

## Worked example

$p=(0.5,0.5)$, $q=(0.9,0.1)$. $H(p)=1$ bit and $H(p,q)=-0.5\log_20.9-0.5\log_20.1=1.737$ bits, so $\mathrm{KL}(p\|q)=0.737$ bits. The other direction gives $\mathrm{KL}(q\|p)=0.9\log_2\tfrac{0.9}{0.5}+0.1\log_2\tfrac{0.1}{0.5}=0.531$ bits — a different number.

**As a loss.** A classifier outputs $q=(0.2,0.7,0.1)$ and the true class is the second: loss $=-\ln0.7=0.357$ nats. Had it output $q_y=0.01$, the loss would be $4.6$ nats — confident mistakes are punished hardest.

## In code

```python
import numpy as np, torch
import torch.nn.functional as F
from scipy.stats import entropy

p, q = np.array([0.5, 0.5]), np.array([0.9, 0.1])
entropy(p, q, base=2)                  # KL(p‖q) = 0.737 bits
entropy(q, p, base=2)                  # KL(q‖p) = 0.531 bits

logits = torch.tensor([[1.2, 2.5, 0.3]])
F.cross_entropy(logits, torch.tensor([1]))      # 0.3245 nats = −log softmax(logits)[1]

P = torch.tensor([[0.5, 0.5]])
F.kl_div(torch.tensor([[0.9, 0.1]]).log(), P, reduction="batchmean")   # KL(P‖Q), nats
```

F.cross_entropy takes raw logits (it applies log-softmax internally); F.kl_div takes log q first and p second, and computes KL(p‖q).

- **Classification:** Every softmax classifier is trained by minimizing cross-entropy with the labels.
- **Language models:** Perplexity $=e^{H}$ for per-token cross-entropy $H$ in nats: $H=2.3$ means the model is as uncertain as a uniform choice among ≈ 10 tokens.
- **VAEs & distillation:** VAEs add $\mathrm{KL}(q(z\mid x)\,\|\,\mathcal N(0,I))$ to regularize the latent space; a distilled student minimizes KL to the teacher's softened outputs.
- **RLHF:** A KL penalty to the reference model keeps a fine-tuned LLM from drifting into degenerate text that games the reward.

> **KL is not a distance:** It is asymmetric and violates the triangle inequality, and which direction you minimize changes the answer — see the Gaussian fit above. When you need a symmetric, bounded measure, use the Jensen–Shannon divergence; for a true metric between distributions, Wasserstein distance.

> **Logs of zero and of softmax:** If $q(x)=0$ where $p(x)>0$, the loss is infinite; label smoothing or a small epsilon prevents it. Never compute `log(softmax(z))` in two steps — it underflows. Pass logits to `F.cross_entropy` or use `log_softmax`, which applies the log-sum-exp trick.

> **Bits or nats?:** Libraries use natural logs. Divide a loss in nats by $\ln 2\approx0.693$ to get bits. The optimum doesn't change — only the units of the reported number.

**Question:** A model gives probability 0.25 to the correct class. Its cross-entropy loss on this example, in bits, is…

- A. 0.25
- B. 1
- C. 2
- D. 4

**Answer:** C. 2 — With a one-hot $p$, $H(p,q)=-\log_2 q_y=-\log_2 0.25=2$ bits.

**Question:** You fit a single Gaussian $q$ to a bimodal $p$ by minimizing $\mathrm{KL}(q\,\|\,p)$. The fit tends to…

- A. spread out to cover both modes
- B. lock onto one mode
- C. match p's mean and variance exactly
- D. reach KL = 0

**Answer:** B. lock onto one mode — Reverse KL averages over $q$: putting mass between the modes, where $p\approx0$, is expensive, while ignoring a mode costs only a bounded amount. Forward KL is the one that matches the moments.

**Key points**

- $H(p,q)=-\sum p\log q$; $\mathrm{KL}(p\|q)=H(p,q)-H(p)\ge0$.
- Cross-entropy loss = negative log-likelihood; gradient w.r.t. logits is q − p.
- KL is asymmetric: forward covers modes, reverse seeks one.
- Feed logits to library losses; watch log(0) and the bits-vs-nats units.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
