Categorical cross-entropy

−Σ q log softmax(z): the multi-class loss. Its logit gradient is p − q, it splits into H(q) + KL(q‖p), and it accepts any target distribution, which is how label smoothing, mixup and distillation work.

11 min read865 wordscore0 viewsupdated 2026-10-11

Definition

Categorical cross-entropy is the loss −∑k=1Kqkln⁡pk-\sum_{k=1}^{K} q_k\ln p_k between a target distribution qq over KK mutually exclusive classes and the model's prediction p=softmax⁡(z)p=\operatorname{softmax}(z). With a one-hot target it reduces to −ln⁡pc-\ln p_c, the negative log-probability of the true class cc.

Softmax plus cross-entropy is the output head of nearly every classifier, from ImageNet CNNs to the next-token prediction of language models. The Loss functions overview derived it as the categorical negative log-likelihood. This article looks at it as a loss between two distributions: what its gradient does to each logit, why it can never reach zero for a soft target, and how replacing the one-hot target with something softer (label smoothing, mixup, distillation) changes what the network learns.

Cross-entropy H(q,p)=−∑kqklog⁡pkH(q,p)=-\sum_k q_k\log p_k is an information-theoretic quantity: the average code length when events from qq are encoded with a code optimized for pp. Its gap above the entropy H(q)H(q) is the Kullback–Leibler divergence (Kullback and Leibler, 1951). Bridle (1990) named the normalized exponential "softmax" and argued for training it with this likelihood criterion instead of squared error.

Formula and gradient

ℓ(z,q)=−∑k=1Kqkln⁡softmax⁡(z)k=log⁡∑jezj−∑kqkzk∂ℓ∂zk=pk−qk\ell(z,q)=-\sum_{k=1}^{K}q_k\ln\operatorname{softmax}(z)_k=\log\sum_{j}e^{z_j}-\sum_k q_kz_k\qquad\frac{\partial\ell}{\partial z_k}=p_k-q_k
z∈RKz\in\mathbb R^K
logits, the raw output of the last linear layer
qq
target distribution: one-hot, label-smoothed or any soft target with ∑kqk=1\sum_k q_k=1
p=softmax⁡(z)p=\operatorname{softmax}(z)
predicted distribution, pk=ezk/∑jezjp_k=e^{z_k}/\sum_j e^{z_j}

The second form is why the gradient is so clean: the derivative of log-sum-exp is softmax, and the linear term contributes −qk-q_k. The gradient needs only ∑kqk=1\sum_k q_k=1, so p−qp-q holds for soft targets too. Its entries sum to 0: raising the true class's logit and lowering the others are the same move. The Hessian is diag⁡(p)−pp⊤\operatorname{diag}(p)-pp^\top, which is positive semidefinite, so the loss is convex in the logits.

H(q,p)=−∑kqkln⁡qk⏟H(q) (fixed by the target)+∑kqkln⁡qkpk⏟KL(q ∥ p) ≥ 0H(q,p)=\underbrace{-\sum_k q_k\ln q_k}_{H(q)\ \text{(fixed by the target)}}+\underbrace{\sum_k q_k\ln\frac{q_k}{p_k}}_{\mathrm{KL}(q\,\|\,p)\ \ge\ 0}
H(q)H(q)
entropy of the target: 0 for one-hot, positive for any soft target
KL(q∥p)\mathrm{KL}(q\|p)
the part the model controls; zero only when p=qp=q

Minimizing cross-entropy over the model is minimizing forward KL from the target, because H(q)H(q) does not depend on zz. Two consequences: the loss floor is H(q)H(q), not 0, for soft targets; and for a one-hot target the floor is reached only in the limit pc→1p_c\to1, i.e. zc−zj→∞z_c-z_j\to\infty.

At a glance

Task
single-label classification with K mutually exclusive classes
Gradient w.r.t. z
p−qp-q; bounded in [−1,1][-1,1] per entry, entries sum to 0
Uniform prediction
ln⁡K\ln K for a one-hot target (2.303 for 10 classes)
Minimum
H(q)H(q) at p=qp=q; for one-hot, approached as the margin →∞\to\infty
Convexity
convex in the logits (Hessian diag⁡(p)−pp⊤⪰0\operatorname{diag}(p)-pp^\top\succeq0)
PyTorch
nn.CrossEntropyLoss(label_smoothing=…); targets may be indices or probabilities
Keras
CategoricalCrossentropy(from_logits=True, label_smoothing=…) with one-hot targets
target q = (1−ε)·onehot + ε/K, ε = 0.00bar p, tick q∂CE/∂zₖ = pₖ − qₖcat ★z=+1.50p=0.578 q=1.000−0.422dogz=+0.80p=0.287 q=0.0000.287foxz=−0.30p=0.096 q=0.0000.096owlz=−1.20p=0.039 q=0.0000.039margin z★ − max otherunder gradient descent (lr 1)04812010100100010⁴step 0 · margin 0.700hard target: optimum at +∞CE = 0.5476 = H(q) 0.0000 + KL(q‖p) 0.5476 nats · Σ(p − q) = 0.000the floor of CE is H(q): a one-hot target has H(q) = 0, so CE → 0 only as p★ → 1
true classtarget
descend
fig.Four classes with logits (1.5, 0.8, −0.3, −1.2). Bars are p, orange ticks the target q, and the middle column is the gradient p − q. With a one-hot target on cat the loss is 0.5476 and the gradient (−0.422, +0.287, +0.096, +0.039). Press the descend button repeatedly: without smoothing the cat margin climbs 0.70 → 3.33 → 5.73 → 8.13 → 10.50 at steps 10, 100, 1000, 10⁴ and never stops. Set ε = 0.10 and the same run levels off at the dashed optimum 3.611, where p = q = (0.925, 0.025, 0.025, 0.025) and CE = H(q) = 0.3488. The 'soft 70/30' target settles at margin ln(7/3) = 0.847, with CE approaching its floor H(q) = 0.6109.

Intuition: what p − q does to each logit

Every class receives a push proportional to how wrong its probability is. The true class gets pc−1<0p_c-1<0, a push up of size "missing probability". Each wrong class gets +pk+p_k, a push down of size "probability it stole". A class the model already ignores (pk≈0p_k\approx0) is left alone, so effort concentrates on the confusable alternatives. When the prediction is right and confident, all entries are small and the example contributes almost nothing, which is the self-paced behaviour that squared error on probabilities lacks.

Label smoothing

With a one-hot target the loss keeps falling as the true logit runs away from the rest, so a network can always reduce training loss by becoming more confident. Label smoothing (Szegedy et al., 2016, introduced with Inception-v3 and ε=0.1\varepsilon=0.1) mixes the one-hot target with the uniform distribution:

qk=(1−ε) 1[k=c]+εK⟹zc∗−zj∗=ln⁡1−ε+ε/Kε/K  (j≠c)q_k=(1-\varepsilon)\,\mathbb 1[k=c]+\frac{\varepsilon}{K}\qquad\Longrightarrow\qquad z_c^*-z_j^*=\ln\frac{1-\varepsilon+\varepsilon/K}{\varepsilon/K}\ \ (j\ne c)
ε\varepsilon
smoothing strength; PyTorch's label_smoothing uses exactly this qq
zc∗−zj∗z_c^*-z_j^*
the logit margin at the optimum p=qp=q: finite for any ε>0\varepsilon>0

The optimum is now a finite margin: ln⁡37=3.611\ln 37=3.611 for K=4K=4, ε=0.1\varepsilon=0.1 (the figure's dashed line), 4.5114.511 for K=10K=10 and 9.1059.105 for K=1000K=1000. Pushing the margin beyond it is penalized, so logits stay bounded. The Transformer (Vaswani et al., 2017) used ε=0.1\varepsilon=0.1, noting it hurts perplexity but improves accuracy and BLEU. Müller, Kornblith and Hinton (2019) showed that it improves calibration and tightens same-class clusters in the penultimate layer, but makes the network a worse teacher for distillation because it erases information about which wrong classes are similar.

Soft targets: mixup and distillation

Because ∂ℓ/∂z=p−q\partial\ell/\partial z=p-q holds for any distribution qq, the same loss trains on soft targets. Mixup (Zhang et al., 2018) blends two images λxa+(1−λ)xb\lambda x_a+(1-\lambda)x_b and their labels λya+(1−λ)yb\lambda y_a+(1-\lambda)y_b. Knowledge distillation (Hinton, Vinyals and Dean, 2015) uses a teacher's temperature-softened softmax as qq, so the student learns the teacher's similarity structure ("this 2 looks a bit like a 7"). In both cases the loss is minimized at p=qp=q and its minimum value is H(q)>0H(q)>0.

a+0.40b+0.18c-0.06d-0.10e-0.10■ p (true)■ q (model)drag bar tops · numbers below = pᵢ log₂(pᵢ/qᵢ)
H(p, q)
2.009 + 0.313 = 2.322
H(q, p)
2.322 + 0.366 = 2.688

■ entropy + ■ KL = cross-entropy (bits). KL(p‖q) = 0.313 vs KL(q‖p) = 0.366 — not symmetric.

fig.From the information-theory chapter: drag p and q and watch cross-entropy split into entropy plus KL. The entropy part is what a soft target adds as an unremovable floor; only the KL part responds to the model.

Worked example

classpq one-hotp − qq, ε = 0.1p − qq, 70/30 mixp − q
cat ★0.57831−0.42170.925−0.34670.7−0.1217
dog0.287200.28720.0250.26220.3−0.0128
fox0.095600.09560.0250.070600.0956
owl0.038900.03890.0250.013900.0389
CE0.54760.67760.7576
H(q) + KL0 + 0.54760.3488 + 0.32880.6109 + 0.1467
Logits z = (1.5, 0.8, −0.3, −1.2), true class cat. Three targets; nats, checked in Node.
  1. 1log⁡∑jezj=2.0476\log\sum_j e^{z_j}=2.0476, so the one-hot loss is 2.0476−1.5=0.5476=−ln⁡0.57832.0476-1.5=0.5476=-\ln0.5783.
  2. 2Smoothed: ℓ=(1−ε)(0.5476)+ε⋅(−ln⁡p)‾=0.9(0.5476)+0.1(1.8476)=0.6776\ell=(1-\varepsilon)(0.5476)+\varepsilon\cdot\overline{(-\ln p)}=0.9(0.5476)+0.1(1.8476)=0.6776, where 1.84761.8476 is the mean of −ln⁡pk-\ln p_k over the four classes. This is PyTorch's formula.
  3. 3Smoothing lowers the push on cat (−0.42→−0.35-0.42\to-0.35) and turns owl's push nearly off: powl=0.039p_{owl}=0.039 is already close to its target 0.025.
  4. 4Under the 70/30 target dog's gradient is negative: pdog=0.287<0.3p_{dog}=0.287<0.3, so the loss wants dog up even though it is not the "true" class. The KL part (0.147) is the only part training can remove.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3z = torch.tensor([[1.5, 0.8, -0.3, -1.2]], requires_grad=True) # logits, shape (N, K)
4c = torch.tensor([0]) # class index
5
6print(nn.CrossEntropyLoss()(z, c)) # 0.5476
7print(nn.CrossEntropyLoss(label_smoothing=0.1)(z, c)) # 0.6776
8
9# soft target: pass probabilities of shape (N, K) instead of indices
10q = torch.tensor([[0.7, 0.3, 0.0, 0.0]])
11loss = nn.CrossEntropyLoss()(z, q) # 0.7576
12loss.backward()
13print(z.grad) # softmax(z) - q: [[-0.1217, -0.0128, 0.0956, 0.0389]]
14
15# the same thing written out: stable log-softmax, then the weighted sum
16manual = -(q * F.log_softmax(z, dim=1)).sum(dim=1).mean()
17
18# distillation: soft teacher targets at temperature T (Hinton et al. 2015)
19T = 2.0
20teacher_logits = torch.tensor([[3.0, 2.0, -1.0, -2.0]])
21q_t = F.softmax(teacher_logits / T, dim=1)
22kd = nn.CrossEntropyLoss()(z / T, q_t) * T**2 # T² keeps gradient scale comparable
Hard, smoothed and soft targets with one PyTorch loss

Use when

  • +Gradient p−qp-q: simple, bounded, non-saturating while wrong, and concentrated on confusable classes
  • +Convex in the logits; calibrated probabilities at the optimum (proper scoring rule)
  • +Works unchanged with soft targets: label smoothing, mixup, distillation
  • +Interpretable: loss in nats, ln⁡K\ln K baseline, perplexity =eloss=e^{\text{loss}} for language models

Avoid when

  • −Unbounded per example: one mislabelled sample can contribute a very large loss
  • −With one-hot targets the optimum is at infinite margin, encouraging over-confidence
  • −Needs mutually exclusive classes; multi-label tasks need per-label BCE
  • −Costs O(K)O(K) per example, which dominates for very large vocabularies

Check yourself

A 1000-class model is trained with label_smoothing = 0.1. Its training loss levels off near 1.0 and will not go lower. What is the most likely explanation?

Check yourself

For logits z and a soft target q = (0.7, 0.3, 0, 0), what is the gradient of the cross-entropy with respect to z?

Remember

  • ✓ℓ=−∑kqkln⁡softmax⁡(z)k=log⁡∑jezj−q⊤z\ell=-\sum_kq_k\ln\operatorname{softmax}(z)_k=\log\sum_je^{z_j}-q^\top z; gradient p−qp-q for any target distribution.
  • ✓H(q,p)=H(q)+KL(q∥p)H(q,p)=H(q)+\mathrm{KL}(q\|p): training removes only the KL part; soft targets leave a floor H(q)H(q).
  • ✓One-hot targets have their optimum at infinite logit margin; label smoothing makes it finite, ln⁡1−ε+ε/Kε/K\ln\frac{1-\varepsilon+\varepsilon/K}{\varepsilon/K}.
  • ✓Label smoothing improves calibration and accuracy in many settings but hurts distillation teachers.
  • ✓PyTorch CrossEntropyLoss takes logits plus indices or probabilities; Keras needs from_logits=True for logits.
← back to Deep Learning roadmap