# Label smoothing

> Train against q = (1 − ε)·onehot + ε/K instead of the one-hot label. The optimal logit gap becomes finite, logits stay bounded and the model is less over-confident, which usually helps accuracy and calibration but makes it a worse teacher for distillation.

Source: https://www.mlfoundry.in/learn/label-smoothing  
Section: Deep Learning › Regularization & Normalization  
Level: core · 10 min read · updated 2026-10-11

**Label smoothing:** trains a $K$-class softmax classifier with cross-entropy against a softened target: the correct class gets probability $1-\varepsilon+\varepsilon/K$ and every class, the correct one included, gets an extra $\varepsilon/K$. The one-hot label is mixed with the uniform distribution, with weight $\varepsilon$ (typically 0.1).

With a one-hot target, cross-entropy $-\ln p_c$ is never zero for finite logits. The network can always lower its training loss by making the correct logit larger relative to the others, so on a separable training set the logits grow without bound and the predicted probabilities head to 1. The model ends up extremely confident on the training set, including on mislabeled or ambiguous examples. Label smoothing puts a ceiling on how confident it is rewarded for being.

Szegedy, Vanhoucke, Ioffe, Shlens and Wojna introduced it in "Rethinking the Inception architecture for computer vision" (2016), as **label-smoothing regularization**, with $K=1000$ and $\varepsilon=0.1$ on ImageNet, reporting about 0.2% absolute improvement in top-1 and top-5 error. The Transformer (Vaswani et al., 2017) used $\varepsilon=0.1$ as well, noting that it hurts perplexity but improves accuracy and BLEU. Müller, Kornblith and Hinton (2019), "When does label smoothing help?", explained what it does to the representation, to calibration and to distillation.

## Formula and gradient

$$
q_k=(1-\varepsilon)\,\mathbb 1[k=c]+\frac{\varepsilon}{K},\qquad \ell=-\sum_{k=1}^K q_k\ln p_k=(1-\varepsilon)\,(-\ln p_c)+\varepsilon\cdot\frac1K\sum_{k=1}^K(-\ln p_k)
$$

- $c$: the labelled class
- $\varepsilon$: smoothing strength, $0\le\varepsilon<1$; PyTorch's `label_smoothing` is exactly this $\varepsilon$
- $p=\mathrm{softmax}(z)$: the model's predicted distribution over the K classes
- $\tfrac1K\sum_k(-\ln p_k)$: cross-entropy to the uniform distribution $u$: equals $\mathrm{KL}(u\,\|\,p)+\ln K$, so it pulls $p$ toward uniform

$$
\frac{\partial\ell}{\partial z_k}=p_k-q_k\qquad\Longrightarrow\qquad \text{at the optimum } p=q:\quad z_c^*-z_j^*=\ln\frac{1-\varepsilon+\varepsilon/K}{\varepsilon/K}\ \ (j\ne c)
$$

- $p_k-q_k$: the usual softmax cross-entropy gradient; it vanishes only when $p=q$
- $z_c^*-z_j^*$: the optimal logit gap: finite for every $\varepsilon>0$; infinite for one-hot targets

The gradient is the same $p-q$ as for any target distribution (see **Categorical cross-entropy**, which derives the gap formula). With $q$ one-hot, the correct class always gets a push $p_c-1<0$ until $p_c=1$, which needs an infinite gap. With smoothing, the push reverses once $p_c$ exceeds $1-\varepsilon+\varepsilon/K$: a model that is more confident than the target is pulled back. For $\varepsilon=0.1$ the optimal gap is $\ln 28=3.332$ for $K=3$, $\ln 37=3.611$ for $K=4$ and $9.105$ for $K=1000$.

**At a glance**

- **Changes:** the target only; the model and the logits are untouched
- **Typical ε:** 0.1 (Inception-v3, Transformer, many ImageNet recipes)
- **Loss floor:** $H(q)$, the entropy of the smoothed target: 0.2911 nats for $K=3$, about 1.01 for $K=1000$ at $\varepsilon=0.1$
- **Equivalent to:** cross-entropy plus $\varepsilon\cdot\mathrm{KL}(u\,\|\,p)$ up to a constant: a confidence penalty
- **PyTorch:** `nn.CrossEntropyLoss(label_smoothing=0.1)`
- **Keras:** `CategoricalCrossentropy(label_smoothing=0.1)` (one-hot targets)

> Figure: Three overlapping classes, 60 training points, a softmax head on 49 Gaussian bump features, 1500 full-batch GD steps; reliability diagram on 600 held-out points (bars: accuracy per confidence bin; red ticks: mean confidence). Without smoothing (grey) the mean training logit gap is still climbing at step 1500 (9.36), held-out confidence is 0.904 against accuracy 0.743, ECE 0.168 and held-out NLL 1.159. With ε = 0.1 the gap levels off below the dashed optimum 3.332 (2.93 at step 1500), confidence drops to 0.823, ECE to 0.096 and NLL to 0.719, while accuracy is about the same (0.735). ε = 0.2 lowers ECE further to 0.067. Scrub back to step 100 with ε = 0.3: confidence 0.682 is now below accuracy 0.773, so too much smoothing makes the model under-confident.

## Worked example: three classes, ε = 0.1

Logits $z=(2,\,0,\,0)$ with the label on class 1. Then $\ln\sum_j e^{z_j}=\ln(e^2+2)=2.2395$, so $p=(0.7870,\,0.1065,\,0.1065)$. The smoothed target is $q=(0.9333,\,0.0333,\,0.0333)$.

1. One-hot loss: $-\ln 0.7870=0.2395$. Gradient $p-\text{onehot}=(-0.2130,\,+0.1065,\,+0.1065)$: push the correct logit up, the others down.
2. Smoothed loss: $(1-\varepsilon)(0.2395)+\varepsilon\cdot\tfrac13(0.2395+2.2395+2.2395)=0.9(0.2395)+0.1(1.5729)=0.3729$. This is what `nn.CrossEntropyLoss(label_smoothing=0.1)` returns.
3. Smoothed gradient $p-q=(-0.1463,\,+0.0732,\,+0.0732)$. Still pushing up, but 31% less, because $p_1=0.787$ is already closer to its target 0.933 than to 1.
4. Split: $\ell=H(q)+\mathrm{KL}(q\,\|\,p)=0.2911+0.0817$. Only the KL part can be trained away; $H(q)=0.2911$ is the floor.

Loss as a function of the gap z₁ − z₂ = z₁ − z₃ (K = 3, ε = 0.1; checked in Node)

| gap | p₁ | one-hot loss | smoothed loss |
| --- | --- | --- | --- |
| 2 | 0.7870 | 0.2395 | 0.3729 |
| 3.332 = ln 28 | 0.9333 | 0.0690 | **0.2911** (minimum) |
| 5 | 0.9867 | 0.0134 | 0.3467 |
| 10 | 0.9999 | 0.0001 | 0.6668 |

The one-hot column falls forever, which is why unsmoothed training keeps inflating the logits. The smoothed column has a minimum at the finite gap 3.332 and *rises* beyond it: at gap 10 the $\varepsilon/K$ mass on the wrong classes costs $\tfrac{0.1}{3}\cdot2\cdot10\approx0.67$ nats. The model is penalized for being more confident than $1-\varepsilon+\varepsilon/K$.

```python
import torch, torch.nn as nn, torch.nn.functional as F

z = torch.tensor([[2.0, 0.0, 0.0]], requires_grad=True)
y = torch.tensor([0])

print(nn.CrossEntropyLoss()(z, y).item())                     # 0.2395
loss = nn.CrossEntropyLoss(label_smoothing=0.1)(z, y)
print(loss.item())                                            # 0.3729
loss.backward()
print(z.grad)                                                 # [[-0.1463, 0.0732, 0.0732]] = p - q

# By hand: q = (1 - eps) * onehot + eps / K
K, eps = 3, 0.1
q = F.one_hot(y, K).float() * (1 - eps) + eps / K
print(-(q * F.log_softmax(z, dim=-1)).sum(-1).mean().item())  # 0.3729

# Training: smooth the loss, but report unsmoothed metrics on validation data
train_loss = nn.CrossEntropyLoss(label_smoothing=0.1)
val_loss = nn.CrossEntropyLoss()
```

Label smoothing in PyTorch, and the same loss by hand

## What it does to the network

**Representations.** Müller et al. (2019) projected penultimate-layer activations and found that label smoothing makes each class a tight, roughly equidistant cluster, because every wrong class must be held at the same logit distance from the correct one. Without smoothing, clusters are broader, and how close an example sits to another class carries information about similarity.

**Calibration.** Over-parameterized networks trained with one-hot cross-entropy tend to be over-confident (Guo et al., 2017). Label smoothing reduces that: Müller et al. found it lowers the expected calibration error by an amount comparable to post-hoc temperature scaling. The figure shows the mechanism in miniature: same accuracy, much less confidence, a lower ECE and a lower held-out NLL. It is not a calibration guarantee. A large $\varepsilon$ makes the model under-confident, and the best $\varepsilon$ for accuracy is not always the best for calibration.

**Distillation.** A teacher trained with label smoothing is often more accurate but a *worse* teacher: students distilled from it do worse than students distilled from a teacher trained without it (Müller et al., 2019). Distillation transfers the teacher's soft probabilities over the wrong classes ("this 3 looks a bit like an 8"), and label smoothing erases exactly that information by forcing all wrong classes to the same small value. **Label noise** is the other common motivation: smoothing limits how hard the model is pushed to fit a wrong label, and Lukasik et al. (2020) found it competitive with dedicated noise-robust losses.

> **Comparing smoothed training loss:** The smoothed loss cannot go below $H(q)$, about 1.0 nats for $K=1000$, $\varepsilon=0.1$. A plateau there is the optimum, not a stuck optimizer. Do not compare a smoothed run's training loss with an unsmoothed one, and report validation accuracy or **unsmoothed** cross-entropy. The same applies to early stopping on validation loss: monitor the plain loss.

> **Smoothing a distillation teacher:** If a model will later serve as a teacher for knowledge distillation, train it without label smoothing (or check that the students do not suffer). The teacher's accuracy can go up while the student's goes down.

> **A confidence penalty in disguise:** Because $\ell=(1-\varepsilon)H(\text{onehot},p)+\varepsilon H(u,p)$ and $H(u,p)=\mathrm{KL}(u\,\|\,p)+\ln K$, label smoothing is cross-entropy plus a penalty that pulls $p$ toward uniform. Pereyra et al. (2017) proposed the closely related **confidence penalty**, which subtracts $\beta H(p)$, the entropy of the prediction, from the loss, i.e. uses $\mathrm{KL}(p\,\|\,u)$ in the other direction. Both regularize the *output* distribution rather than the weights.

> **Soft targets in general:** Label smoothing is the simplest soft target. **Mixup** and **CutMix** produce soft targets by mixing two examples, and knowledge distillation uses a teacher's softmax. All of them use the same loss and the same gradient $p-q$; they differ only in where $q$ comes from.

**Strengths**

- One argument in the loss; no change to the model or the cost of training
- Bounds the logits: a finite optimal gap instead of an incentive to grow forever
- Usually a small accuracy gain on large classification tasks, and better calibration than one-hot training
- Limits the damage from mislabeled examples

**Limitations**

- Hurts knowledge distillation: erases the similarity structure among wrong classes
- Too much smoothing makes the model under-confident
- Training loss has a floor H(q), which confuses loss-based comparisons and monitoring
- Treats all wrong classes as equally likely, which is rarely true

**Question:** With K = 10 classes and ε = 0.1, what probability does the smoothed target give the correct class?

- A. 0.90
- B. 0.91
- C. 0.99
- D. 0.10

**Answer:** B. 0.91 — $q_c=1-\varepsilon+\varepsilon/K=0.9+0.01=0.91$, and each of the 9 wrong classes gets $\varepsilon/K=0.01$. The optimal logit gap is $\ln(0.91/0.01)=4.511$.

**Question:** A team trains a large teacher with label smoothing ε = 0.1; its accuracy improves. Students distilled from it are worse than students from the unsmoothed teacher. Why?

- A. Label smoothing lowers the teacher's accuracy
- B. The teacher's soft probabilities over the wrong classes carry less information: smoothing pushes all of them toward the same small value
- C. Distillation cannot use soft targets
- D. The student needs label smoothing too

**Answer:** B. The teacher's soft probabilities over the wrong classes carry less information: smoothing pushes all of them toward the same small value — Distillation works by transferring the teacher's relative probabilities among the wrong classes, which encode which classes look alike. Label smoothing trains every wrong class toward the same ε/K and tightens each class into an isolated cluster, so that information is largely erased (Müller, Kornblith and Hinton, 2019).

**Key points**

- Target $q=(1-\varepsilon)\,\text{onehot}+\varepsilon/K$; loss $(1-\varepsilon)(-\ln p_c)+\varepsilon\cdot\overline{(-\ln p_k)}$; gradient $p-q$.
- The optimal logit gap is finite, $\ln\frac{1-\varepsilon+\varepsilon/K}{\varepsilon/K}$ (3.332 for $K=3$, 9.105 for $K=1000$ at $\varepsilon=0.1$), so logits stay bounded.
- The training loss has a floor $H(q)$; monitor accuracy or unsmoothed cross-entropy.
- It usually improves accuracy and calibration and tightens class clusters, but too large an ε makes the model under-confident.
- It makes the network a worse distillation teacher, because it erases the similarity information among wrong classes.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
