Definition
Categorical cross-entropy is the loss between a target distribution over mutually exclusive classes and the model's prediction . With a one-hot target it reduces to , the negative log-probability of the true class .
Softmax plus cross-entropy is the output head of nearly every classifier, from ImageNet CNNs to the next-token prediction of language models. The Loss functions overview derived it as the categorical negative log-likelihood. This article looks at it as a loss between two distributions: what its gradient does to each logit, why it can never reach zero for a soft target, and how replacing the one-hot target with something softer (label smoothing, mixup, distillation) changes what the network learns.
Cross-entropy is an information-theoretic quantity: the average code length when events from are encoded with a code optimized for . Its gap above the entropy is the Kullback–Leibler divergence (Kullback and Leibler, 1951). Bridle (1990) named the normalized exponential "softmax" and argued for training it with this likelihood criterion instead of squared error.
Formula and gradient
- logits, the raw output of the last linear layer
- target distribution: one-hot, label-smoothed or any soft target with
- predicted distribution,
The second form is why the gradient is so clean: the derivative of log-sum-exp is softmax, and the linear term contributes . The gradient needs only , so holds for soft targets too. Its entries sum to 0: raising the true class's logit and lowering the others are the same move. The Hessian is , which is positive semidefinite, so the loss is convex in the logits.
- entropy of the target: 0 for one-hot, positive for any soft target
- the part the model controls; zero only when
Minimizing cross-entropy over the model is minimizing forward KL from the target, because does not depend on . Two consequences: the loss floor is , not 0, for soft targets; and for a one-hot target the floor is reached only in the limit , i.e. .
At a glance
- Task
- single-label classification with K mutually exclusive classes
- Gradient w.r.t. z
- ; bounded in per entry, entries sum to 0
- Uniform prediction
- for a one-hot target (2.303 for 10 classes)
- Minimum
- at ; for one-hot, approached as the margin
- Convexity
- convex in the logits (Hessian )
- PyTorch
nn.CrossEntropyLoss(label_smoothing=…); targets may be indices or probabilities- Keras
CategoricalCrossentropy(from_logits=True, label_smoothing=…)with one-hot targets
Intuition: what p − q does to each logit
Every class receives a push proportional to how wrong its probability is. The true class gets , a push up of size "missing probability". Each wrong class gets , a push down of size "probability it stole". A class the model already ignores () is left alone, so effort concentrates on the confusable alternatives. When the prediction is right and confident, all entries are small and the example contributes almost nothing, which is the self-paced behaviour that squared error on probabilities lacks.
Label smoothing
With a one-hot target the loss keeps falling as the true logit runs away from the rest, so a network can always reduce training loss by becoming more confident. Label smoothing (Szegedy et al., 2016, introduced with Inception-v3 and ) mixes the one-hot target with the uniform distribution:
- smoothing strength; PyTorch's
label_smoothinguses exactly this - the logit margin at the optimum : finite for any
The optimum is now a finite margin: for , (the figure's dashed line), for and for . Pushing the margin beyond it is penalized, so logits stay bounded. The Transformer (Vaswani et al., 2017) used , noting it hurts perplexity but improves accuracy and BLEU. Müller, Kornblith and Hinton (2019) showed that it improves calibration and tightens same-class clusters in the penultimate layer, but makes the network a worse teacher for distillation because it erases information about which wrong classes are similar.
Soft targets: mixup and distillation
Because holds for any distribution , the same loss trains on soft targets. Mixup (Zhang et al., 2018) blends two images and their labels . Knowledge distillation (Hinton, Vinyals and Dean, 2015) uses a teacher's temperature-softened softmax as , so the student learns the teacher's similarity structure ("this 2 looks a bit like a 7"). In both cases the loss is minimized at and its minimum value is .
■ entropy + ■ KL = cross-entropy (bits). KL(p‖q) = 0.313 vs KL(q‖p) = 0.366 — not symmetric.
Worked example
| class | p | q one-hot | p − q | q, ε = 0.1 | p − q | q, 70/30 mix | p − q |
|---|---|---|---|---|---|---|---|
| cat ★ | 0.5783 | 1 | −0.4217 | 0.925 | −0.3467 | 0.7 | −0.1217 |
| dog | 0.2872 | 0 | 0.2872 | 0.025 | 0.2622 | 0.3 | −0.0128 |
| fox | 0.0956 | 0 | 0.0956 | 0.025 | 0.0706 | 0 | 0.0956 |
| owl | 0.0389 | 0 | 0.0389 | 0.025 | 0.0139 | 0 | 0.0389 |
| CE | 0.5476 | 0.6776 | 0.7576 | ||||
| H(q) + KL | 0 + 0.5476 | 0.3488 + 0.3288 | 0.6109 + 0.1467 |
- 1, so the one-hot loss is .
- 2Smoothed: , where is the mean of over the four classes. This is PyTorch's formula.
- 3Smoothing lowers the push on cat () and turns owl's push nearly off: is already close to its target 0.025.
- 4Under the 70/30 target dog's gradient is negative: , so the loss wants dog up even though it is not the "true" class. The KL part (0.147) is the only part training can remove.
1import torch, torch.nn as nn, torch.nn.functional as F23z = torch.tensor([[1.5, 0.8, -0.3, -1.2]], requires_grad=True) # logits, shape (N, K)4c = torch.tensor([0]) # class index56print(nn.CrossEntropyLoss()(z, c)) # 0.54767print(nn.CrossEntropyLoss(label_smoothing=0.1)(z, c)) # 0.677689# soft target: pass probabilities of shape (N, K) instead of indices10q = torch.tensor([[0.7, 0.3, 0.0, 0.0]])11loss = nn.CrossEntropyLoss()(z, q) # 0.757612loss.backward()13print(z.grad) # softmax(z) - q: [[-0.1217, -0.0128, 0.0956, 0.0389]]1415# the same thing written out: stable log-softmax, then the weighted sum16manual = -(q * F.log_softmax(z, dim=1)).sum(dim=1).mean()1718# distillation: soft teacher targets at temperature T (Hinton et al. 2015)19T = 2.020teacher_logits = torch.tensor([[3.0, 2.0, -1.0, -2.0]])21q_t = F.softmax(teacher_logits / T, dim=1)22kd = nn.CrossEntropyLoss()(z / T, q_t) * T**2 # T² keeps gradient scale comparableUse when
- +Gradient : simple, bounded, non-saturating while wrong, and concentrated on confusable classes
- +Convex in the logits; calibrated probabilities at the optimum (proper scoring rule)
- +Works unchanged with soft targets: label smoothing, mixup, distillation
- +Interpretable: loss in nats, baseline, perplexity for language models
Avoid when
- −Unbounded per example: one mislabelled sample can contribute a very large loss
- −With one-hot targets the optimum is at infinite margin, encouraging over-confidence
- −Needs mutually exclusive classes; multi-label tasks need per-label BCE
- −Costs per example, which dominates for very large vocabularies
Check yourself
A 1000-class model is trained with label_smoothing = 0.1. Its training loss levels off near 1.0 and will not go lower. What is the most likely explanation?
Check yourself
For logits z and a soft target q = (0.7, 0.3, 0, 0), what is the gradient of the cross-entropy with respect to z?
Remember
- ✓; gradient for any target distribution.
- ✓: training removes only the KL part; soft targets leave a floor .
- ✓One-hot targets have their optimum at infinite logit margin; label smoothing makes it finite, .
- ✓Label smoothing improves calibration and accuracy in many settings but hurts distillation teachers.
- ✓PyTorch
CrossEntropyLosstakes logits plus indices or probabilities; Keras needsfrom_logits=Truefor logits.