Cross-Entropy & KL Divergence

H(p, q) = H(p) + KL(p‖q), so minimizing cross-entropy over q minimizes the KL divergence to p.

6 min read456 wordscore4 viewsupdated 2026-10-03

Definition

Cross-entropy H(p,q)=−∑xp(x)log⁡q(x)H(p,q)=-\sum_x p(x)\log q(x) is the average cost of encoding outcomes drawn from pp with a code designed for qq. The Kullback–Leibler divergence KL(p ∥ q)=H(p,q)−H(p)\mathrm{KL}(p\,\|\,q)=H(p,q)-H(p) is the extra cost of using the wrong distribution: zero when q=pq=p, positive otherwise.

At a glance

Notation
H(p,q)H(p,q); DKL(p ∥ q)D_{\mathrm{KL}}(p\,\|\,q) — read "KL of pp from qq"
Units
bits with log₂, nats with ln; deep-learning libraries use nats
Key property
KL(p∥q)≥0\mathrm{KL}(p\|q)\ge0, with equality iff p=qp=q (Gibbs' inequality)
Not a distance
KL(p∥q)≠KL(q∥p)\mathrm{KL}(p\|q)\ne\mathrm{KL}(q\|p) in general; no triangle inequality
Prerequisites
entropy, expectation, logarithms
Used in
classification loss, language-model perplexity, VAEs, distillation, RLHF

The intuition

Outcomes come from the true distribution pp, but your model believes qq. Each outcome xx costs −log⁡q(x)-\log q(x) bits under your model — a lot when the model thought xx was unlikely. Averaged over the real outcomes, you pay H(p,q)H(p,q). The best anyone could pay is H(p)H(p), reached only by the true distribution. The gap, KL, is the price of being wrong.

The math

H(p,q)=Ex∼p[−log⁡q(x)],KL(p ∥ q)=∑xp(x)log⁡p(x)q(x)=H(p,q)−H(p)H(p,q)=\mathbb{E}_{x\sim p}\big[-\log q(x)\big],\qquad \mathrm{KL}(p\,\|\,q)=\sum_x p(x)\log\frac{p(x)}{q(x)}=H(p,q)-H(p)
pp
true (data) distribution
qq
model distribution

Gibbs' inequality. Over the outcomes where p>0p>0, Jensen's inequality for the concave log⁡\log gives −KL(p∥q)=∑plog⁡qp≤log⁡∑p qp=log⁡∑q≤log⁡1=0-\mathrm{KL}(p\|q)=\sum p\log\frac{q}{p}\le\log\sum p\,\frac{q}{p}=\log\sum q\le\log1=0. So KL≥0\mathrm{KL}\ge0, with equality only when q=pq=p.

Why it is the classification loss. For a labelled example, pp is one-hot on the true class yy, so H(p,q)=−log⁡qyH(p,q)=-\log q_y: the negative log-likelihood of the correct class. Averaged over the data, minimizing cross-entropy is maximum likelihood. And because H(p)H(p) does not depend on the model, minimizing cross-entropy and minimizing KL to the data are the same optimization. With q=softmax(z)q=\mathrm{softmax}(z), the gradient with respect to the logits is simply:

∂H(p,q)∂z=q−p\frac{\partial H(p,q)}{\partial z}=q-p

Asymmetry. KL(p∥q)\mathrm{KL}(p\|q) averages over pp, so it is infinite wherever p>0p>0 but q=0q=0: the model must cover everything the data does (mode-covering). KL(q∥p)\mathrm{KL}(q\|p) averages over qq, so it punishes putting mass where p≈0p\approx0 and ignores modes qq skips (mode-seeking).

Gaussians have a closed form, in nats — the KL term of every VAE uses it with N(0,1)\mathcal N(0,1) as the prior:

KL(N(μ1,σ12) ∥ N(μ2,σ22))=log⁡σ2σ1+σ12+(μ1−μ2)22σ22−12\mathrm{KL}\big(\mathcal N(\mu_1,\sigma_1^2)\,\|\,\mathcal N(\mu_2,\sigma_2^2)\big)=\log\frac{\sigma_2}{\sigma_1}+\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2}-\frac12

See it

a+0.40b+0.18c-0.06d-0.10e-0.10■ p (true)■ q (model)drag bar tops · numbers below = pᵢ log₂(pᵢ/qᵢ)
H(p, q)
2.009 + 0.313 = 2.322
H(q, p)
2.322 + 0.366 = 2.688

■ entropy + ■ KL = cross-entropy (bits). KL(p‖q) = 0.313 vs KL(q‖p) = 0.366 — not symmetric.

fig.Tab 1: drag p and q. Bars show cross-entropy = entropy + KL in both directions; "q misses e" makes KL(p‖q) infinite. Tab 2: fitting one Gaussian to a bimodal p — forward KL covers both modes (σ = 2.09, KL(p‖q) = 0.80 bits), reverse KL locks onto one (KL(q‖p) = 1.00 bit).

Worked example

p=(0.5,0.5)p=(0.5,0.5), q=(0.9,0.1)q=(0.9,0.1). H(p)=1H(p)=1 bit and H(p,q)=−0.5log⁡20.9−0.5log⁡20.1=1.737H(p,q)=-0.5\log_20.9-0.5\log_20.1=1.737 bits, so KL(p∥q)=0.737\mathrm{KL}(p\|q)=0.737 bits. The other direction gives KL(q∥p)=0.9log⁡20.90.5+0.1log⁡20.10.5=0.531\mathrm{KL}(q\|p)=0.9\log_2\tfrac{0.9}{0.5}+0.1\log_2\tfrac{0.1}{0.5}=0.531 bits — a different number.

As a loss. A classifier outputs q=(0.2,0.7,0.1)q=(0.2,0.7,0.1) and the true class is the second: loss =−ln⁡0.7=0.357=-\ln0.7=0.357 nats. Had it output qy=0.01q_y=0.01, the loss would be 4.64.6 nats — confident mistakes are punished hardest.

In code

python
1import numpy as np, torch
2import torch.nn.functional as F
3from scipy.stats import entropy
4
5p, q = np.array([0.5, 0.5]), np.array([0.9, 0.1])
6entropy(p, q, base=2) # KL(p‖q) = 0.737 bits
7entropy(q, p, base=2) # KL(q‖p) = 0.531 bits
8
9logits = torch.tensor([[1.2, 2.5, 0.3]])
10F.cross_entropy(logits, torch.tensor([1])) # 0.3245 nats = −log softmax(logits)[1]
11
12P = torch.tensor([[0.5, 0.5]])
13F.kl_div(torch.tensor([[0.9, 0.1]]).log(), P, reduction="batchmean") # KL(P‖Q), nats
F.cross_entropy takes raw logits (it applies log-softmax internally); F.kl_div takes log q first and p second, and computes KL(p‖q).

Where it's used

Classification

Every softmax classifier is trained by minimizing cross-entropy with the labels.

Language models

Perplexity =eH=e^{H} for per-token cross-entropy HH in nats: H=2.3H=2.3 means the model is as uncertain as a uniform choice among ≈ 10 tokens.

VAEs & distillation

VAEs add KL(q(z∣x) ∥ N(0,I))\mathrm{KL}(q(z\mid x)\,\|\,\mathcal N(0,I)) to regularize the latent space; a distilled student minimizes KL to the teacher's softened outputs.

RLHF

A KL penalty to the reference model keeps a fine-tuned LLM from drifting into degenerate text that games the reward.

Check yourself

A model gives probability 0.25 to the correct class. Its cross-entropy loss on this example, in bits, is…

Check yourself

You fit a single Gaussian qq to a bimodal pp by minimizing KL(q ∥ p)\mathrm{KL}(q\,\|\,p). The fit tends to…

Remember

  • ✓H(p,q)=−∑plog⁡qH(p,q)=-\sum p\log q; KL(p∥q)=H(p,q)−H(p)≥0\mathrm{KL}(p\|q)=H(p,q)-H(p)\ge0.
  • ✓Cross-entropy loss = negative log-likelihood; gradient w.r.t. logits is q − p.
  • ✓KL is asymmetric: forward covers modes, reverse seeks one.
  • ✓Feed logits to library losses; watch log(0) and the bits-vs-nats units.
← back to Mathematics roadmap