Definition
Cross-entropy is the average cost of encoding outcomes drawn from with a code designed for . The Kullback–Leibler divergence is the extra cost of using the wrong distribution: zero when , positive otherwise.
At a glance
- Notation
- ; — read "KL of from "
- Units
- bits with log₂, nats with ln; deep-learning libraries use nats
- Key property
- , with equality iff (Gibbs' inequality)
- Not a distance
- in general; no triangle inequality
- Prerequisites
- entropy, expectation, logarithms
- Used in
- classification loss, language-model perplexity, VAEs, distillation, RLHF
The intuition
Outcomes come from the true distribution , but your model believes . Each outcome costs bits under your model — a lot when the model thought was unlikely. Averaged over the real outcomes, you pay . The best anyone could pay is , reached only by the true distribution. The gap, KL, is the price of being wrong.
The math
- true (data) distribution
- model distribution
Gibbs' inequality. Over the outcomes where , Jensen's inequality for the concave gives . So , with equality only when .
Why it is the classification loss. For a labelled example, is one-hot on the true class , so : the negative log-likelihood of the correct class. Averaged over the data, minimizing cross-entropy is maximum likelihood. And because does not depend on the model, minimizing cross-entropy and minimizing KL to the data are the same optimization. With , the gradient with respect to the logits is simply:
Asymmetry. averages over , so it is infinite wherever but : the model must cover everything the data does (mode-covering). averages over , so it punishes putting mass where and ignores modes skips (mode-seeking).
Gaussians have a closed form, in nats — the KL term of every VAE uses it with as the prior:
See it
■ entropy + ■ KL = cross-entropy (bits). KL(p‖q) = 0.313 vs KL(q‖p) = 0.366 — not symmetric.
Worked example
, . bit and bits, so bits. The other direction gives bits — a different number.
As a loss. A classifier outputs and the true class is the second: loss nats. Had it output , the loss would be nats — confident mistakes are punished hardest.
In code
1import numpy as np, torch2import torch.nn.functional as F3from scipy.stats import entropy45p, q = np.array([0.5, 0.5]), np.array([0.9, 0.1])6entropy(p, q, base=2) # KL(p‖q) = 0.737 bits7entropy(q, p, base=2) # KL(q‖p) = 0.531 bits89logits = torch.tensor([[1.2, 2.5, 0.3]])10F.cross_entropy(logits, torch.tensor([1])) # 0.3245 nats = −log softmax(logits)[1]1112P = torch.tensor([[0.5, 0.5]])13F.kl_div(torch.tensor([[0.9, 0.1]]).log(), P, reduction="batchmean") # KL(P‖Q), natsWhere it's used
Classification
Every softmax classifier is trained by minimizing cross-entropy with the labels.
Language models
Perplexity for per-token cross-entropy in nats: means the model is as uncertain as a uniform choice among ≈ 10 tokens.
VAEs & distillation
VAEs add to regularize the latent space; a distilled student minimizes KL to the teacher's softened outputs.
RLHF
A KL penalty to the reference model keeps a fine-tuned LLM from drifting into degenerate text that games the reward.
Check yourself
A model gives probability 0.25 to the correct class. Its cross-entropy loss on this example, in bits, is…
Check yourself
You fit a single Gaussian to a bimodal by minimizing . The fit tends to…
Remember
- ✓; .
- ✓Cross-entropy loss = negative log-likelihood; gradient w.r.t. logits is q − p.
- ✓KL is asymmetric: forward covers modes, reverse seeks one.
- ✓Feed logits to library losses; watch log(0) and the bits-vs-nats units.