# Sparse categorical cross-entropy

> The same loss as categorical cross-entropy, but the target is a class index instead of a one-hot vector: the loss is one gather, −log_softmax(z)[y]. It saves K-fold target memory and lets padding be skipped with ignore_index.

Source: https://www.mlfoundry.in/learn/sparse-categorical-cross-entropy  
Section: Deep Learning › Loss Functions  
Level: core · 9 min read · updated 2026-10-11

**Sparse categorical cross-entropy:** is categorical cross-entropy with the target given as an integer class index $y\in\{0,\dots,K-1\}$ rather than a one-hot vector: $\ell=-\ln\operatorname{softmax}(z)_y=\log\sum_je^{z_j}-z_y$. "Sparse" refers to the label encoding, not to the model or the loss values.

A one-hot vector has $K-1$ zeros and a single 1, so $-\sum_kq_k\ln p_k$ picks out one term. Storing and multiplying all the zeros is wasted work. The sparse form stores just the index and **gathers** the one log-probability it needs. The numbers are identical; what changes is memory, speed, the API you call, and a few practical features such as skipping padding tokens.

The name comes from Keras, which exposes the two encodings as separate losses: `CategoricalCrossentropy` for one-hot targets and `SparseCategoricalCrossentropy` for integer targets. PyTorch has a single `nn.CrossEntropyLoss` that dispatches on the target: an integer tensor of shape $(N,)$ means class indices (the sparse form), a float tensor of shape $(N,K)$ means probabilities.

## Formula: a gather instead of a dot product

$$
\ell_i=-\log\operatorname{softmax}(z_i)_{y_i}=\log\sum_{k}e^{z_{ik}}-z_{i,y_i},\qquad \mathcal L=\frac{1}{|\{i:y_i\ne\text{ign}\}|}\sum_{i:\,y_i\ne\text{ign}}\ell_i
$$

- $z_i\in\mathbb R^K$: logits of example (or token) $i$
- $y_i$: integer class index, or the ignore value (PyTorch default $-100$)
- $\text{ign}$: `ignore_index`: such positions contribute no loss, no gradient, and are not counted in the mean

$$
\frac{\partial\ell_i}{\partial z_{ik}}=p_{ik}-\mathbb 1[k=y_i]
$$

- $p_{ik}$: softmax probability; the gradient is $p$ with 1 subtracted at the label position, a scatter, no one-hot tensor built

**At a glance**

- **Same as:** categorical cross-entropy with a one-hot target; values and gradients are identical
- **Target:** shape $(N,)$, integer; PyTorch requires `torch.long` (int64)
- **Target memory:** $8N$ bytes (int64) instead of $4NK$ bytes (float32 one-hot)
- **Cannot express:** soft targets (mixup, distillation); label smoothing is still available as a flag
- **PyTorch:** `nn.CrossEntropyLoss(ignore_index=-100)` on logits; `nn.NLLLoss` on log-probabilities
- **Keras:** `SparseCategoricalCrossentropy(from_logits=True, ignore_class=None)`

> Figure: A six-token batch over a five-word vocabulary. Each row is log_softmax of that token's logits; the orange outline is the single cell the integer label gathers. Toggle 'one-hot' to see the dense target: the per-token losses are identical (0.327, 0.503, 0.734, 0.937). Tokens 4 and 5 are padding (label −100): with ignore_index the batch loss is 0.6251, the mean over the 4 real tokens. If the padding were labelled 0 and averaged in, the loss would read 0.7257. Click any label to change it. The memory line compares target storage: for 8×2048 tokens and GPT-2's 50,257-token vocabulary, one-hot float32 needs 3.29 GB while int64 labels need 131 KB, a factor of K/2, about 25,000.

## Why it matters: memory at large K

For 10-class MNIST the encoding is irrelevant: a batch of 256 needs 10.2 KB of one-hot floats versus 2 KB of indices. For language models it is decisive. A batch of 8 sequences of 2048 tokens is $N=16{,}384$ targets. With GPT-2's vocabulary of $K=50{,}257$ the one-hot tensor is $16{,}384\times50{,}257\times4$ bytes $=3.29$ GB; with Llama 3's 128,256-token vocabulary it is 8.41 GB. The integer labels take 131 KB in both cases.

The logits themselves are also $N\times K$ floats, so a dense target would roughly double the memory of the output layer and add an $N\times K$ multiply. The sparse path reads one logit per row for the loss and writes the gradient as $p$ with a scalar subtracted at one position.

## ignore_index: padding and masked positions

Sequence batches are padded to a common length, and many objectives score only some positions (masked language modelling scores the masked tokens; instruction tuning often scores only the response). In the sparse form you mark those positions with a sentinel label. PyTorch's `ignore_index` (default $-100$, which is why Hugging Face sets unused labels to $-100$) skips them **and** removes them from the denominator of the mean. Recent Keras versions offer `ignore_class` for the same purpose.

> **Padding as a real class:** If padding is labelled with a real index (often 0) and not ignored, the model is trained to predict "pad" after the end of every sequence, and the reported loss mixes real tokens with trivially easy or impossible pad predictions. In the figure that changes the batch loss from 0.6251 to 0.7257. A one-hot encoding cannot mark "no target" at all except by an all-zero row, which some code then divides by the wrong count.

> **Mean with ignore_index and weights:** With `reduction="mean"`, PyTorch divides by the number of non-ignored targets, or with class `weight` by the sum of their weights. When gradient accumulation splits a batch into micro-batches with different numbers of real tokens, averaging the micro-batch means is not the same as the token mean of the full batch; sum with `reduction="sum"` and divide by the total token count instead.

## Worked example

Per-token loss = −log_softmax(z)[y] for the figure's batch (checked numerically in Node)

| token | label y | log_softmax(z)[y] | loss | counted? |
| --- | --- | --- | --- | --- |
| 0 | 0 (the) | −0.3272 | 0.3272 | yes |
| 1 | 1 (cat) | −0.5027 | 0.5027 | yes |
| 2 | 2 (sat) | −0.7337 | 0.7337 | yes |
| 3 | 3 (on) | −0.9366 | 0.9366 | yes |
| 4 | −100 (pad) | – | 0 | no |
| 5 | −100 (pad) | – | 0 | no |

1. Sum over counted tokens $=0.3272+0.5027+0.7337+0.9366=2.5002$; mean $=2.5002/4=0.6251$.
2. The dense version builds a $6\times5$ one-hot matrix and computes $-\sum_k q_{ik}\ln p_{ik}$ per row: 25 of the 30 products are $0\times\ln p$, and the result is the same four numbers.
3. Bug version: label pads as 0, giving extra losses $1.0708$ and $0.7830$; the mean over 6 tokens becomes $4.3539/6=0.7257$.
4. Gradient for token 0: $p_0$ with 1 subtracted at index 0, i.e. $(e^{-0.3272}-1,\,e^{-2.2272},\dots)=(-0.279,\,0.108,\,0.054,\,0.088,\,0.029)$. Pad rows get exactly zero gradient.

```python
import torch, torch.nn as nn, torch.nn.functional as F

logits = torch.tensor([[ 2.2,  0.3, -0.4,  0.1, -1.0],
                       [ 0.1,  1.9,  0.4, -0.6,  0.2],
                       [-0.5,  0.6,  1.4,  0.2, -0.3],
                       [ 0.4, -0.2,  0.3,  1.1,  0.0],
                       [ 0.9,  0.2, -0.7,  0.3,  0.5],
                       [ 1.2, -0.4,  0.1, -0.2,  0.3]])
labels = torch.tensor([0, 1, 2, 3, -100, -100])      # int64; -100 = padding

print(F.cross_entropy(logits, labels))                # 0.6251 (mean over 4 tokens)

# identical to the dense form on the real tokens
keep = labels != -100
onehot = F.one_hot(labels[keep], num_classes=5).float()
print(F.cross_entropy(logits[keep], onehot))          # 0.6251

# language model shapes: logits (B, T, V), labels (B, T)
B, T, V = 2, 7, 100
lm_logits = torch.randn(B, T, V)
lm_labels = torch.randint(0, V, (B, T))
loss = F.cross_entropy(lm_logits.reshape(-1, V), lm_labels.reshape(-1))
# or keep the sequence axis, but the class axis must be dim 1: (B, V, T)
loss_same = F.cross_entropy(lm_logits.transpose(1, 2), lm_labels)

# Keras / TensorFlow equivalent (class axis last):
# keras.losses.SparseCategoricalCrossentropy(from_logits=True)(labels, logits)
```

Integer targets, sequence shapes and ignore_index in PyTorch (and the Keras equivalent)

> **Shape and dtype errors:** PyTorch wants the class axis in **dimension 1**: $(N,K)$ or $(N,K,d_1,\dots)$ with targets $(N,d_1,\dots)$. Passing $(B,T,V)$ logits directly treats $T$ as the class axis and fails or silently trains the wrong thing when $T=V$. Integer targets must be `torch.long`; a float tensor of shape $(N,K)$ is read as probabilities. A label outside $[0,K)$ that is not the ignore value raises an index error on CPU and a device-side assert on CUDA.

> **Sparse is the natural form of a likelihood:** Maximum likelihood asks for $-\ln p(y_i\mid x_i)$ at the **observed** label. The integer form states that directly; the one-hot form is a roundabout way of selecting the same term. A one-hot vector only earns its memory when the target is genuinely a distribution, as in mixup or distillation.

**Strengths**

- Same loss and gradient as categorical cross-entropy at a fraction of the target memory ($K/2$ times less than float32 one-hot)
- No one-hot materialization: a gather forward, a scatter backward
- `ignore_index` cleanly drops padding and unscored positions from loss, gradient and denominator
- Matches the data: labels usually arrive as integers

**Limitations**

- Cannot represent soft or multi-label targets
- Easy to get shapes wrong (PyTorch class axis is dim 1, Keras is last)
- Out-of-range labels crash, sometimes with an opaque CUDA assert
- A sentinel value that differs between libraries ($-100$ in PyTorch, `ignore_class` in Keras)

**Question:** You switch a classifier from one-hot targets with CategoricalCrossentropy to integer targets with SparseCategoricalCrossentropy (same model, same data). What happens to the loss values?

- A. They are identical
- B. They drop by a factor of K
- C. They increase because the loss now ignores the wrong classes
- D. They become probabilities instead of nats

**Answer:** A. They are identical — $-\sum_k\mathbb 1[k=y]\ln p_k=-\ln p_y$ exactly. The wrong classes still matter through the softmax normalizer $\log\sum_je^{z_j}$; only the storage of the target changed.

**Question:** A batch has 3 real tokens with losses 0.2, 0.4, 0.9 and 5 padding tokens labelled −100. What does F.cross_entropy(..., ignore_index=-100) return?

- A. 0.1875
- B. 0.5
- C. 1.5
- D. It raises an error because −100 is out of range

**Answer:** B. 0.5 — Ignored positions contribute nothing and are removed from the count: $(0.2+0.4+0.9)/3=0.5$. Dividing by all 8 positions (0.1875) is what a hand-written masked sum gets wrong.

**Key points**

- Sparse CCE is categorical cross-entropy with an integer label: $\ell=\log\sum_je^{z_j}-z_y$, gradient $p-\mathrm{onehot}(y)$ via a scatter.
- Keras: `SparseCategoricalCrossentropy` vs `CategoricalCrossentropy`. PyTorch: one `CrossEntropyLoss` that reads int64 $(N,)$ targets as indices and float $(N,K)$ as probabilities.
- Target memory drops from $4NK$ to $8N$ bytes: 3.29 GB vs 131 KB for 16,384 tokens and a 50,257-word vocabulary.
- `ignore_index=-100` removes padding from the loss, the gradient and the mean's denominator.
- PyTorch puts the class axis at dim 1; flatten (B, T, V) to (B·T, V) or transpose to (B, V, T).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
