# Categorical cross-entropy

> −Σ q log softmax(z): the multi-class loss. Its logit gradient is p − q, it splits into H(q) + KL(q‖p), and it accepts any target distribution, which is how label smoothing, mixup and distillation work.

Source: https://www.mlfoundry.in/learn/categorical-cross-entropy  
Section: Deep Learning › Loss Functions  
Level: core · 11 min read · updated 2026-10-11

**Categorical cross-entropy:** is the loss $-\sum_{k=1}^{K} q_k\ln p_k$ between a target distribution $q$ over $K$ mutually exclusive classes and the model's prediction $p=\operatorname{softmax}(z)$. With a one-hot target it reduces to $-\ln p_c$, the negative log-probability of the true class $c$.

Softmax plus cross-entropy is the output head of nearly every classifier, from ImageNet CNNs to the next-token prediction of language models. The **Loss functions** overview derived it as the categorical negative log-likelihood. This article looks at it as a loss between two **distributions**: what its gradient does to each logit, why it can never reach zero for a soft target, and how replacing the one-hot target with something softer (label smoothing, mixup, distillation) changes what the network learns.

Cross-entropy $H(q,p)=-\sum_k q_k\log p_k$ is an information-theoretic quantity: the average code length when events from $q$ are encoded with a code optimized for $p$. Its gap above the entropy $H(q)$ is the Kullback–Leibler divergence (Kullback and Leibler, 1951). Bridle (1990) named the normalized exponential "softmax" and argued for training it with this likelihood criterion instead of squared error.

## Formula and gradient

$$
\ell(z,q)=-\sum_{k=1}^{K}q_k\ln\operatorname{softmax}(z)_k=\log\sum_{j}e^{z_j}-\sum_k q_kz_k\qquad\frac{\partial\ell}{\partial z_k}=p_k-q_k
$$

- $z\in\mathbb R^K$: logits, the raw output of the last linear layer
- $q$: target distribution: one-hot, label-smoothed or any soft target with $\sum_k q_k=1$
- $p=\operatorname{softmax}(z)$: predicted distribution, $p_k=e^{z_k}/\sum_j e^{z_j}$

The second form is why the gradient is so clean: the derivative of log-sum-exp is softmax, and the linear term contributes $-q_k$. The gradient needs only $\sum_k q_k=1$, so $p-q$ holds for soft targets too. Its entries sum to 0: raising the true class's logit and lowering the others are the same move. The Hessian is $\operatorname{diag}(p)-pp^\top$, which is positive semidefinite, so the loss is convex in the logits.

$$
H(q,p)=\underbrace{-\sum_k q_k\ln q_k}_{H(q)\ \text{(fixed by the target)}}+\underbrace{\sum_k q_k\ln\frac{q_k}{p_k}}_{\mathrm{KL}(q\,\|\,p)\ \ge\ 0}
$$

- $H(q)$: entropy of the target: 0 for one-hot, positive for any soft target
- $\mathrm{KL}(q\|p)$: the part the model controls; zero only when $p=q$

Minimizing cross-entropy over the model is minimizing forward KL from the target, because $H(q)$ does not depend on $z$. Two consequences: the loss floor is $H(q)$, not 0, for soft targets; and for a one-hot target the floor is reached only in the limit $p_c\to1$, i.e. $z_c-z_j\to\infty$.

**At a glance**

- **Task:** single-label classification with K mutually exclusive classes
- **Gradient w.r.t. z:** $p-q$; bounded in $[-1,1]$ per entry, entries sum to 0
- **Uniform prediction:** $\ln K$ for a one-hot target (2.303 for 10 classes)
- **Minimum:** $H(q)$ at $p=q$; for one-hot, approached as the margin $\to\infty$
- **Convexity:** convex in the logits (Hessian $\operatorname{diag}(p)-pp^\top\succeq0$)
- **PyTorch:** `nn.CrossEntropyLoss(label_smoothing=…)`; targets may be indices or probabilities
- **Keras:** `CategoricalCrossentropy(from_logits=True, label_smoothing=…)` with one-hot targets

> Figure: Four classes with logits (1.5, 0.8, −0.3, −1.2). Bars are p, orange ticks the target q, and the middle column is the gradient p − q. With a one-hot target on cat the loss is 0.5476 and the gradient (−0.422, +0.287, +0.096, +0.039). Press the descend button repeatedly: without smoothing the cat margin climbs 0.70 → 3.33 → 5.73 → 8.13 → 10.50 at steps 10, 100, 1000, 10⁴ and never stops. Set ε = 0.10 and the same run levels off at the dashed optimum 3.611, where p = q = (0.925, 0.025, 0.025, 0.025) and CE = H(q) = 0.3488. The 'soft 70/30' target settles at margin ln(7/3) = 0.847, with CE approaching its floor H(q) = 0.6109.

## Intuition: what p − q does to each logit

Every class receives a push proportional to how wrong its probability is. The true class gets $p_c-1<0$, a push up of size "missing probability". Each wrong class gets $+p_k$, a push down of size "probability it stole". A class the model already ignores ($p_k\approx0$) is left alone, so effort concentrates on the confusable alternatives. When the prediction is right and confident, all entries are small and the example contributes almost nothing, which is the self-paced behaviour that squared error on probabilities lacks.

## Label smoothing

With a one-hot target the loss keeps falling as the true logit runs away from the rest, so a network can always reduce training loss by becoming more confident. **Label smoothing** (Szegedy et al., 2016, introduced with Inception-v3 and $\varepsilon=0.1$) mixes the one-hot target with the uniform distribution:

$$
q_k=(1-\varepsilon)\,\mathbb 1[k=c]+\frac{\varepsilon}{K}\qquad\Longrightarrow\qquad z_c^*-z_j^*=\ln\frac{1-\varepsilon+\varepsilon/K}{\varepsilon/K}\ \ (j\ne c)
$$

- $\varepsilon$: smoothing strength; PyTorch's `label_smoothing` uses exactly this $q$
- $z_c^*-z_j^*$: the logit margin at the optimum $p=q$: finite for any $\varepsilon>0$

The optimum is now a finite margin: $\ln 37=3.611$ for $K=4$, $\varepsilon=0.1$ (the figure's dashed line), $4.511$ for $K=10$ and $9.105$ for $K=1000$. Pushing the margin beyond it is penalized, so logits stay bounded. The Transformer (Vaswani et al., 2017) used $\varepsilon=0.1$, noting it hurts perplexity but improves accuracy and BLEU. Müller, Kornblith and Hinton (2019) showed that it improves calibration and tightens same-class clusters in the penultimate layer, but makes the network a worse teacher for distillation because it erases information about which wrong classes are similar.

> **The loss has a floor:** With smoothing, the training loss cannot go below $H(q)$: for $K=4$, $\varepsilon=0.1$ that is 0.3488 nats; for $K=1000$, $\varepsilon=0.1$ it is about 1.0. A plateau at that value is the optimum, not a stuck optimizer. Compare models by accuracy or by unsmoothed cross-entropy on validation data, not by the smoothed training loss.

## Soft targets: mixup and distillation

Because $\partial\ell/\partial z=p-q$ holds for any distribution $q$, the same loss trains on soft targets. **Mixup** (Zhang et al., 2018) blends two images $\lambda x_a+(1-\lambda)x_b$ and their labels $\lambda y_a+(1-\lambda)y_b$. **Knowledge distillation** (Hinton, Vinyals and Dean, 2015) uses a teacher's temperature-softened softmax as $q$, so the student learns the teacher's similarity structure ("this 2 looks a bit like a 7"). In both cases the loss is minimized at $p=q$ and its minimum value is $H(q)>0$.

> Figure: From the information-theory chapter: drag p and q and watch cross-entropy split into entropy plus KL. The entropy part is what a soft target adds as an unremovable floor; only the KL part responds to the model.

## Worked example

Logits z = (1.5, 0.8, −0.3, −1.2), true class cat. Three targets; nats, checked in Node.

| class | p | q one-hot | p − q | q, ε = 0.1 | p − q | q, 70/30 mix | p − q |
| --- | --- | --- | --- | --- | --- | --- | --- |
| cat ★ | 0.5783 | 1 | −0.4217 | 0.925 | −0.3467 | 0.7 | −0.1217 |
| dog | 0.2872 | 0 | 0.2872 | 0.025 | 0.2622 | 0.3 | −0.0128 |
| fox | 0.0956 | 0 | 0.0956 | 0.025 | 0.0706 | 0 | 0.0956 |
| owl | 0.0389 | 0 | 0.0389 | 0.025 | 0.0139 | 0 | 0.0389 |
| **CE** |  | **0.5476** |  | **0.6776** |  | **0.7576** |  |
| H(q) + KL |  | 0 + 0.5476 |  | 0.3488 + 0.3288 |  | 0.6109 + 0.1467 |  |

1. $\log\sum_j e^{z_j}=2.0476$, so the one-hot loss is $2.0476-1.5=0.5476=-\ln0.5783$.
2. Smoothed: $\ell=(1-\varepsilon)(0.5476)+\varepsilon\cdot\overline{(-\ln p)}=0.9(0.5476)+0.1(1.8476)=0.6776$, where $1.8476$ is the mean of $-\ln p_k$ over the four classes. This is PyTorch's formula.
3. Smoothing lowers the push on cat ($-0.42\to-0.35$) and turns owl's push nearly off: $p_{owl}=0.039$ is already close to its target 0.025.
4. Under the 70/30 target dog's gradient is negative: $p_{dog}=0.287<0.3$, so the loss wants dog *up* even though it is not the "true" class. The KL part (0.147) is the only part training can remove.

```python
import torch, torch.nn as nn, torch.nn.functional as F

z = torch.tensor([[1.5, 0.8, -0.3, -1.2]], requires_grad=True)   # logits, shape (N, K)
c = torch.tensor([0])                                            # class index

print(nn.CrossEntropyLoss()(z, c))                       # 0.5476
print(nn.CrossEntropyLoss(label_smoothing=0.1)(z, c))    # 0.6776

# soft target: pass probabilities of shape (N, K) instead of indices
q = torch.tensor([[0.7, 0.3, 0.0, 0.0]])
loss = nn.CrossEntropyLoss()(z, q)                       # 0.7576
loss.backward()
print(z.grad)                    # softmax(z) - q: [[-0.1217, -0.0128, 0.0956, 0.0389]]

# the same thing written out: stable log-softmax, then the weighted sum
manual = -(q * F.log_softmax(z, dim=1)).sum(dim=1).mean()

# distillation: soft teacher targets at temperature T (Hinton et al. 2015)
T = 2.0
teacher_logits = torch.tensor([[3.0, 2.0, -1.0, -2.0]])
q_t = F.softmax(teacher_logits / T, dim=1)
kd = nn.CrossEntropyLoss()(z / T, q_t) * T**2           # T² keeps gradient scale comparable
```

Hard, smoothed and soft targets with one PyTorch loss

> **Keras naming:** Keras splits by target format: `CategoricalCrossentropy` expects one-hot (or soft) targets of shape $(N,K)$, while `SparseCategoricalCrossentropy` takes integer labels. Both default to `from_logits=False`, i.e. they expect probabilities; set `from_logits=True` when the model outputs raw logits. PyTorch's `CrossEntropyLoss` always expects logits and accepts either target format.

> **Cross-entropy is a proper scoring rule:** If the true conditional distribution of the label is $q$, the expected loss $\mathbb E_{y\sim q}[-\ln p_y]=H(q,p)$ is minimized exactly at $p=q$. That is why a well-fit softmax model outputs calibrated probabilities, and why label smoothing (which changes $q$) changes calibration.

**Strengths**

- Gradient $p-q$: simple, bounded, non-saturating while wrong, and concentrated on confusable classes
- Convex in the logits; calibrated probabilities at the optimum (proper scoring rule)
- Works unchanged with soft targets: label smoothing, mixup, distillation
- Interpretable: loss in nats, $\ln K$ baseline, perplexity $=e^{\text{loss}}$ for language models

**Limitations**

- Unbounded per example: one mislabelled sample can contribute a very large loss
- With one-hot targets the optimum is at infinite margin, encouraging over-confidence
- Needs mutually exclusive classes; multi-label tasks need per-label BCE
- Costs $O(K)$ per example, which dominates for very large vocabularies

**Question:** A 1000-class model is trained with label_smoothing = 0.1. Its training loss levels off near 1.0 and will not go lower. What is the most likely explanation?

- A. The learning rate is too high
- B. The loss has reached its floor H(q), the entropy of the smoothed target
- C. The model is underfitting and needs more layers
- D. Label smoothing makes the gradient zero

**Answer:** B. The loss has reached its floor H(q), the entropy of the smoothed target — $q$ puts $0.9+0.0001$ on the true class and $0.0001$ on each of the 999 others. Its entropy is about $1.0$ nats, and $H(q,p)=H(q)+\mathrm{KL}(q\|p)\ge H(q)$. The plateau means $p\approx q$: the model has fit the smoothed target.

**Question:** For logits z and a soft target q = (0.7, 0.3, 0, 0), what is the gradient of the cross-entropy with respect to z?

- A. softmax(z) − q
- B. −q / softmax(z)
- C. softmax(z) − onehot(argmax q)
- D. q − softmax(z)

**Answer:** A. softmax(z) − q — $\ell=\log\sum_j e^{z_j}-\sum_kq_kz_k$, so $\partial\ell/\partial z_k=p_k-q_k$. This only requires $\sum_kq_k=1$. The option $-q/p$ is the gradient with respect to $p$, before chaining through softmax.

**Key points**

- $\ell=-\sum_kq_k\ln\operatorname{softmax}(z)_k=\log\sum_je^{z_j}-q^\top z$; gradient $p-q$ for any target distribution.
- $H(q,p)=H(q)+\mathrm{KL}(q\|p)$: training removes only the KL part; soft targets leave a floor $H(q)$.
- One-hot targets have their optimum at infinite logit margin; label smoothing makes it finite, $\ln\frac{1-\varepsilon+\varepsilon/K}{\varepsilon/K}$.
- Label smoothing improves calibration and accuracy in many settings but hurts distillation teachers.
- PyTorch `CrossEntropyLoss` takes logits plus indices or probabilities; Keras needs `from_logits=True` for logits.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
