Definition
Sparse categorical cross-entropy is categorical cross-entropy with the target given as an integer class index rather than a one-hot vector: . "Sparse" refers to the label encoding, not to the model or the loss values.
A one-hot vector has zeros and a single 1, so picks out one term. Storing and multiplying all the zeros is wasted work. The sparse form stores just the index and gathers the one log-probability it needs. The numbers are identical; what changes is memory, speed, the API you call, and a few practical features such as skipping padding tokens.
The name comes from Keras, which exposes the two encodings as separate losses: CategoricalCrossentropy for one-hot targets and SparseCategoricalCrossentropy for integer targets. PyTorch has a single nn.CrossEntropyLoss that dispatches on the target: an integer tensor of shape means class indices (the sparse form), a float tensor of shape means probabilities.
Formula: a gather instead of a dot product
- logits of example (or token)
- integer class index, or the ignore value (PyTorch default )
ignore_index: such positions contribute no loss, no gradient, and are not counted in the mean
- softmax probability; the gradient is with 1 subtracted at the label position, a scatter, no one-hot tensor built
At a glance
- Same as
- categorical cross-entropy with a one-hot target; values and gradients are identical
- Target
- shape , integer; PyTorch requires
torch.long(int64) - Target memory
- bytes (int64) instead of bytes (float32 one-hot)
- Cannot express
- soft targets (mixup, distillation); label smoothing is still available as a flag
- PyTorch
nn.CrossEntropyLoss(ignore_index=-100)on logits;nn.NLLLosson log-probabilities- Keras
SparseCategoricalCrossentropy(from_logits=True, ignore_class=None)
Why it matters: memory at large K
For 10-class MNIST the encoding is irrelevant: a batch of 256 needs 10.2 KB of one-hot floats versus 2 KB of indices. For language models it is decisive. A batch of 8 sequences of 2048 tokens is targets. With GPT-2's vocabulary of the one-hot tensor is bytes GB; with Llama 3's 128,256-token vocabulary it is 8.41 GB. The integer labels take 131 KB in both cases.
The logits themselves are also floats, so a dense target would roughly double the memory of the output layer and add an multiply. The sparse path reads one logit per row for the loss and writes the gradient as with a scalar subtracted at one position.
ignore_index: padding and masked positions
Sequence batches are padded to a common length, and many objectives score only some positions (masked language modelling scores the masked tokens; instruction tuning often scores only the response). In the sparse form you mark those positions with a sentinel label. PyTorch's ignore_index (default , which is why Hugging Face sets unused labels to ) skips them and removes them from the denominator of the mean. Recent Keras versions offer ignore_class for the same purpose.
Worked example
| token | label y | log_softmax(z)[y] | loss | counted? |
|---|---|---|---|---|
| 0 | 0 (the) | −0.3272 | 0.3272 | yes |
| 1 | 1 (cat) | −0.5027 | 0.5027 | yes |
| 2 | 2 (sat) | −0.7337 | 0.7337 | yes |
| 3 | 3 (on) | −0.9366 | 0.9366 | yes |
| 4 | −100 (pad) | – | 0 | no |
| 5 | −100 (pad) | – | 0 | no |
- 1Sum over counted tokens ; mean .
- 2The dense version builds a one-hot matrix and computes per row: 25 of the 30 products are , and the result is the same four numbers.
- 3Bug version: label pads as 0, giving extra losses and ; the mean over 6 tokens becomes .
- 4Gradient for token 0: with 1 subtracted at index 0, i.e. . Pad rows get exactly zero gradient.
1import torch, torch.nn as nn, torch.nn.functional as F23logits = torch.tensor([[ 2.2, 0.3, -0.4, 0.1, -1.0],4 [ 0.1, 1.9, 0.4, -0.6, 0.2],5 [-0.5, 0.6, 1.4, 0.2, -0.3],6 [ 0.4, -0.2, 0.3, 1.1, 0.0],7 [ 0.9, 0.2, -0.7, 0.3, 0.5],8 [ 1.2, -0.4, 0.1, -0.2, 0.3]])9labels = torch.tensor([0, 1, 2, 3, -100, -100]) # int64; -100 = padding1011print(F.cross_entropy(logits, labels)) # 0.6251 (mean over 4 tokens)1213# identical to the dense form on the real tokens14keep = labels != -10015onehot = F.one_hot(labels[keep], num_classes=5).float()16print(F.cross_entropy(logits[keep], onehot)) # 0.62511718# language model shapes: logits (B, T, V), labels (B, T)19B, T, V = 2, 7, 10020lm_logits = torch.randn(B, T, V)21lm_labels = torch.randint(0, V, (B, T))22loss = F.cross_entropy(lm_logits.reshape(-1, V), lm_labels.reshape(-1))23# or keep the sequence axis, but the class axis must be dim 1: (B, V, T)24loss_same = F.cross_entropy(lm_logits.transpose(1, 2), lm_labels)2526# Keras / TensorFlow equivalent (class axis last):27# keras.losses.SparseCategoricalCrossentropy(from_logits=True)(labels, logits)Use when
- +Same loss and gradient as categorical cross-entropy at a fraction of the target memory ( times less than float32 one-hot)
- +No one-hot materialization: a gather forward, a scatter backward
- +
ignore_indexcleanly drops padding and unscored positions from loss, gradient and denominator - +Matches the data: labels usually arrive as integers
Avoid when
- −Cannot represent soft or multi-label targets
- −Easy to get shapes wrong (PyTorch class axis is dim 1, Keras is last)
- −Out-of-range labels crash, sometimes with an opaque CUDA assert
- −A sentinel value that differs between libraries ( in PyTorch,
ignore_classin Keras)
Check yourself
You switch a classifier from one-hot targets with CategoricalCrossentropy to integer targets with SparseCategoricalCrossentropy (same model, same data). What happens to the loss values?
Check yourself
A batch has 3 real tokens with losses 0.2, 0.4, 0.9 and 5 padding tokens labelled −100. What does F.cross_entropy(..., ignore_index=-100) return?
Remember
- ✓Sparse CCE is categorical cross-entropy with an integer label: , gradient via a scatter.
- ✓Keras:
SparseCategoricalCrossentropyvsCategoricalCrossentropy. PyTorch: oneCrossEntropyLossthat reads int64 targets as indices and float as probabilities. - ✓Target memory drops from to bytes: 3.29 GB vs 131 KB for 16,384 tokens and a 50,257-word vocabulary.
- ✓
ignore_index=-100removes padding from the loss, the gradient and the mean's denominator. - ✓PyTorch puts the class axis at dim 1; flatten (B, T, V) to (B·T, V) or transpose to (B, V, T).