Sparse categorical cross-entropy

The same loss as categorical cross-entropy, but the target is a class index instead of a one-hot vector: the loss is one gather, −log_softmax(z)[y]. It saves K-fold target memory and lets padding be skipped with ignore_index.

9 min read762 wordscore0 viewsupdated 2026-10-11

Definition

Sparse categorical cross-entropy is categorical cross-entropy with the target given as an integer class index y∈{0,…,K−1}y\in\{0,\dots,K-1\} rather than a one-hot vector: ℓ=−ln⁡softmax⁡(z)y=log⁡∑jezj−zy\ell=-\ln\operatorname{softmax}(z)_y=\log\sum_je^{z_j}-z_y. "Sparse" refers to the label encoding, not to the model or the loss values.

A one-hot vector has K−1K-1 zeros and a single 1, so −∑kqkln⁡pk-\sum_kq_k\ln p_k picks out one term. Storing and multiplying all the zeros is wasted work. The sparse form stores just the index and gathers the one log-probability it needs. The numbers are identical; what changes is memory, speed, the API you call, and a few practical features such as skipping padding tokens.

The name comes from Keras, which exposes the two encodings as separate losses: CategoricalCrossentropy for one-hot targets and SparseCategoricalCrossentropy for integer targets. PyTorch has a single nn.CrossEntropyLoss that dispatches on the target: an integer tensor of shape (N,)(N,) means class indices (the sparse form), a float tensor of shape (N,K)(N,K) means probabilities.

Formula: a gather instead of a dot product

ℓi=−log⁡softmax⁡(zi)yi=log⁡∑kezik−zi,yi,L=1∣{i:yi≠ign}∣∑i: yi≠ignℓi\ell_i=-\log\operatorname{softmax}(z_i)_{y_i}=\log\sum_{k}e^{z_{ik}}-z_{i,y_i},\qquad \mathcal L=\frac{1}{|\{i:y_i\ne\text{ign}\}|}\sum_{i:\,y_i\ne\text{ign}}\ell_i
zi∈RKz_i\in\mathbb R^K
logits of example (or token) ii
yiy_i
integer class index, or the ignore value (PyTorch default −100-100)
ign\text{ign}
ignore_index: such positions contribute no loss, no gradient, and are not counted in the mean
∂ℓi∂zik=pik−1[k=yi]\frac{\partial\ell_i}{\partial z_{ik}}=p_{ik}-\mathbb 1[k=y_i]
pikp_{ik}
softmax probability; the gradient is pp with 1 subtracted at the label position, a scatter, no one-hot tensor built

At a glance

Same as
categorical cross-entropy with a one-hot target; values and gradients are identical
Target
shape (N,)(N,), integer; PyTorch requires torch.long (int64)
Target memory
8N8N bytes (int64) instead of 4NK4NK bytes (float32 one-hot)
Cannot express
soft targets (mixup, distillation); label smoothing is still available as a flag
PyTorch
nn.CrossEntropyLoss(ignore_index=-100) on logits; nn.NLLLoss on log-probabilities
Keras
SparseCategoricalCrossentropy(from_logits=True, ignore_class=None)
loss_i = −log_softmax(z_i)[y_i] · click a label to change it (… → -100 = padding)log_softmax(z) per token0:the1:cat2:sat3:on4:matlossinteger target (1 int64)tok 0−0.33−2.23−2.93−2.43−3.530.3270= thetok 1−2.30−0.50−2.00−3.00−2.200.5031= cattok 2−2.63−1.53−0.73−1.93−2.430.7342= sattok 3−1.64−2.24−1.74−0.94−2.040.9373= ontok 4−1.07−1.77−2.67−1.67−1.47–-100ignoredtok 5−0.78−2.38−1.88−2.18−1.68–-100ignoredmean with ignore_index = 0.6251 over 4 tokensbug: 2 pad tokens labelled 0 and averaged in = 0.7257 over 6targets for N = 16,384 tokens, K = 50,257 classes:one-hot float32 3.29 GB · int64 labels 131 KB · ratio K/2 = 25,129×
targets as
presets
fig.A six-token batch over a five-word vocabulary. Each row is log_softmax of that token's logits; the orange outline is the single cell the integer label gathers. Toggle 'one-hot' to see the dense target: the per-token losses are identical (0.327, 0.503, 0.734, 0.937). Tokens 4 and 5 are padding (label −100): with ignore_index the batch loss is 0.6251, the mean over the 4 real tokens. If the padding were labelled 0 and averaged in, the loss would read 0.7257. Click any label to change it. The memory line compares target storage: for 8×2048 tokens and GPT-2's 50,257-token vocabulary, one-hot float32 needs 3.29 GB while int64 labels need 131 KB, a factor of K/2, about 25,000.

Why it matters: memory at large K

For 10-class MNIST the encoding is irrelevant: a batch of 256 needs 10.2 KB of one-hot floats versus 2 KB of indices. For language models it is decisive. A batch of 8 sequences of 2048 tokens is N=16,384N=16{,}384 targets. With GPT-2's vocabulary of K=50,257K=50{,}257 the one-hot tensor is 16,384×50,257×416{,}384\times50{,}257\times4 bytes =3.29=3.29 GB; with Llama 3's 128,256-token vocabulary it is 8.41 GB. The integer labels take 131 KB in both cases.

The logits themselves are also N×KN\times K floats, so a dense target would roughly double the memory of the output layer and add an N×KN\times K multiply. The sparse path reads one logit per row for the loss and writes the gradient as pp with a scalar subtracted at one position.

ignore_index: padding and masked positions

Sequence batches are padded to a common length, and many objectives score only some positions (masked language modelling scores the masked tokens; instruction tuning often scores only the response). In the sparse form you mark those positions with a sentinel label. PyTorch's ignore_index (default −100-100, which is why Hugging Face sets unused labels to −100-100) skips them and removes them from the denominator of the mean. Recent Keras versions offer ignore_class for the same purpose.

Worked example

tokenlabel ylog_softmax(z)[y]losscounted?
00 (the)−0.32720.3272yes
11 (cat)−0.50270.5027yes
22 (sat)−0.73370.7337yes
33 (on)−0.93660.9366yes
4−100 (pad)–0no
5−100 (pad)–0no
Per-token loss = −log_softmax(z)[y] for the figure's batch (checked numerically in Node)
  1. 1Sum over counted tokens =0.3272+0.5027+0.7337+0.9366=2.5002=0.3272+0.5027+0.7337+0.9366=2.5002; mean =2.5002/4=0.6251=2.5002/4=0.6251.
  2. 2The dense version builds a 6×56\times5 one-hot matrix and computes −∑kqikln⁡pik-\sum_k q_{ik}\ln p_{ik} per row: 25 of the 30 products are 0×ln⁡p0\times\ln p, and the result is the same four numbers.
  3. 3Bug version: label pads as 0, giving extra losses 1.07081.0708 and 0.78300.7830; the mean over 6 tokens becomes 4.3539/6=0.72574.3539/6=0.7257.
  4. 4Gradient for token 0: p0p_0 with 1 subtracted at index 0, i.e. (e−0.3272−1, e−2.2272,… )=(−0.279, 0.108, 0.054, 0.088, 0.029)(e^{-0.3272}-1,\,e^{-2.2272},\dots)=(-0.279,\,0.108,\,0.054,\,0.088,\,0.029). Pad rows get exactly zero gradient.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3logits = torch.tensor([[ 2.2, 0.3, -0.4, 0.1, -1.0],
4 [ 0.1, 1.9, 0.4, -0.6, 0.2],
5 [-0.5, 0.6, 1.4, 0.2, -0.3],
6 [ 0.4, -0.2, 0.3, 1.1, 0.0],
7 [ 0.9, 0.2, -0.7, 0.3, 0.5],
8 [ 1.2, -0.4, 0.1, -0.2, 0.3]])
9labels = torch.tensor([0, 1, 2, 3, -100, -100]) # int64; -100 = padding
10
11print(F.cross_entropy(logits, labels)) # 0.6251 (mean over 4 tokens)
12
13# identical to the dense form on the real tokens
14keep = labels != -100
15onehot = F.one_hot(labels[keep], num_classes=5).float()
16print(F.cross_entropy(logits[keep], onehot)) # 0.6251
17
18# language model shapes: logits (B, T, V), labels (B, T)
19B, T, V = 2, 7, 100
20lm_logits = torch.randn(B, T, V)
21lm_labels = torch.randint(0, V, (B, T))
22loss = F.cross_entropy(lm_logits.reshape(-1, V), lm_labels.reshape(-1))
23# or keep the sequence axis, but the class axis must be dim 1: (B, V, T)
24loss_same = F.cross_entropy(lm_logits.transpose(1, 2), lm_labels)
25
26# Keras / TensorFlow equivalent (class axis last):
27# keras.losses.SparseCategoricalCrossentropy(from_logits=True)(labels, logits)
Integer targets, sequence shapes and ignore_index in PyTorch (and the Keras equivalent)

Use when

  • +Same loss and gradient as categorical cross-entropy at a fraction of the target memory (K/2K/2 times less than float32 one-hot)
  • +No one-hot materialization: a gather forward, a scatter backward
  • +ignore_index cleanly drops padding and unscored positions from loss, gradient and denominator
  • +Matches the data: labels usually arrive as integers

Avoid when

  • −Cannot represent soft or multi-label targets
  • −Easy to get shapes wrong (PyTorch class axis is dim 1, Keras is last)
  • −Out-of-range labels crash, sometimes with an opaque CUDA assert
  • −A sentinel value that differs between libraries (−100-100 in PyTorch, ignore_class in Keras)

Check yourself

You switch a classifier from one-hot targets with CategoricalCrossentropy to integer targets with SparseCategoricalCrossentropy (same model, same data). What happens to the loss values?

Check yourself

A batch has 3 real tokens with losses 0.2, 0.4, 0.9 and 5 padding tokens labelled −100. What does F.cross_entropy(..., ignore_index=-100) return?

Remember

  • ✓Sparse CCE is categorical cross-entropy with an integer label: ℓ=log⁡∑jezj−zy\ell=\log\sum_je^{z_j}-z_y, gradient p−onehot(y)p-\mathrm{onehot}(y) via a scatter.
  • ✓Keras: SparseCategoricalCrossentropy vs CategoricalCrossentropy. PyTorch: one CrossEntropyLoss that reads int64 (N,)(N,) targets as indices and float (N,K)(N,K) as probabilities.
  • ✓Target memory drops from 4NK4NK to 8N8N bytes: 3.29 GB vs 131 KB for 16,384 tokens and a 50,257-word vocabulary.
  • ✓ignore_index=-100 removes padding from the loss, the gradient and the mean's denominator.
  • ✓PyTorch puts the class axis at dim 1; flatten (B, T, V) to (B·T, V) or transpose to (B, V, T).
← back to Deep Learning roadmap