# DistilBERT

> DistilBERT (Sanh et al., 2019) is a 6-layer BERT distilled during pre-training with a triple loss: soft-target cross-entropy at temperature T, masked LM and a hidden-state cosine loss. Initialised from every other teacher layer, it is 40% smaller, 60% faster and keeps 97% of BERT-base's GLUE score.

Source: https://www.mlfoundry.in/learn/distilbert  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 14 min read · updated 2026-10-13

**DistilBERT:** is the compressed BERT of Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf (Hugging Face), "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter" (arXiv, October 2019; NeurIPS 2019 EMC² workshop). A 6-layer student with BERT-base's width is trained to imitate the 12-layer BERT-base teacher **during pre-training**, not just on one downstream task. The objective is a sum of three terms: a distillation cross-entropy against the teacher's temperature-softened output distribution, the usual masked-language-model loss, and a cosine loss that aligns the student's and teacher's final hidden states. The student starts from the teacher's weights, taking one layer out of two.

By 2019 pre-trained language models were growing fast: BERT-large had 340M parameters, and newer models went well beyond that. Running them on a CPU, a phone or under a tight latency budget was costly. **Knowledge distillation** (Buciluǎ et al. 2006; Hinton, Vinyals and Dean 2015) offered a fix: train a small *student* to reproduce the outputs of a large *teacher*. Earlier work on BERT had distilled task-specific models, for example into BiLSTMs for one classification task. DistilBERT applied distillation to the **general-purpose** model instead. The student is distilled once, on the pre-training corpus, and can then be fine-tuned on any task like BERT itself.

The architecture follows from one observation in the paper. For a fixed parameter budget, the hidden size has a smaller effect on computation efficiency than the number of layers, because the matrix multiplications along the last dimension are highly optimised. So the authors kept $d=768$ and 12 heads and **halved the depth**. They also removed the token-type (segment) embeddings and the pooler, which only served next-sentence prediction and the [CLS] classifier.

## Architecture and initialisation

> Figure: Teacher and student side by side. Each student layer is drawn level with the teacher layer whose weights it starts from: one layer out of two, with the released extraction script copying teacher layers 1, 3, 5, 8, 10 and 12. Both models produce logits over the 30,522-token vocabulary at every position. The teacher's logits z_t and the student's z_s are softened by the same temperature T for the distillation loss. The student's unsoftened logits feed the masked-LM loss, and the two last hidden states feed the cosine loss. The teacher is frozen; only the student is trained.

The student is a BERT encoder with the same embeddings, attention blocks and feed-forward blocks, but 6 layers instead of 12. Because the shapes match, its weights can be copied directly from the teacher. The paper initialises it by "taking one layer out of two", which gives the student a starting point that already computes useful contextual features. The token and position embeddings and the MLM head are copied too. The ablation below shows this initialisation is the single most important ingredient.

## The triple loss

$$
t_i=\frac{\exp(z^{t}_i/T)}{\sum_j\exp(z^{t}_j/T)},\qquad s_i=\frac{\exp(z^{s}_i/T)}{\sum_j\exp(z^{s}_j/T)},\qquad L_{ce}=-\sum_i t_i\log s_i
$$

- $z^t,\ z^s$: teacher and student logits over the vocabulary at one position
- $T$: softmax temperature, applied to **both** models in training; $T=1$ at inference
- $t_i$: soft target: the teacher's probability for word i
- $L_{ce}$: distillation cross-entropy; it equals $\mathrm{KL}(t\,\|\,s)+H(t)$, and $H(t)$ does not depend on the student

The soft targets carry what Hinton called **dark knowledge**. For "the cat sat on the [MASK]", a teacher that puts 65% on *mat* still ranks *floor* far above *car*. A one-hot label throws that ranking away. Raising $T$ flattens the distribution so that the small probabilities, and therefore the ranking among wrong answers, contribute more to the loss.

$$
\frac{\partial\,\mathrm{KL}(t\|s)}{\partial z^s_i}=\frac{1}{T}\,(s_i-t_i)\qquad\Longrightarrow\qquad \frac{\partial\,T^2\mathrm{KL}(t\|s)}{\partial z^s_i}=T\,(s_i-t_i)
$$

- $1/T$: the chain rule through $z/T$; and since $s-t$ itself shrinks roughly like $1/T$ for large $T$, the raw gradient falls like $1/T^2$
- $T^2$: Hinton et al.'s correction, used in the Hugging Face distiller, so the soft term keeps its weight relative to the hard MLM term when $T$ changes

$$
L_{mlm}=-\log\operatorname{softmax}(z^s)_{y},\qquad L_{cos}=1-\cos\big(h^s,h^t\big),\qquad L=\alpha_{ce}L_{ce}+\alpha_{mlm}L_{mlm}+\alpha_{cos}L_{cos}
$$

- $y$: the original token at a masked position (BERT's MLM, at T = 1)
- $h^s,\ h^t$: last-layer hidden states of student and teacher at the same position, both 768-dimensional
- $L_{cos}$: aligns the directions of the hidden-state vectors, not their lengths
- $\alpha$: the paper gives no values; the example command in the released training code uses $\alpha_{ce}=5$, $\alpha_{mlm}=2$, $\alpha_{cos}=1$

Each term does a different job. $L_{ce}$ transfers the teacher's full output distribution. $L_{mlm}$ keeps the student anchored to the real data, which matters where the teacher is wrong. $L_{cos}$ adds a signal at the representation level: the hidden states, not just the predictions, should point the same way. The cosine form leaves the norm free, so the student does not have to match the teacher's scale exactly.

**DistilBERT in the paper**

- **Teacher:** BERT-base: 12 layers, $d=768$, 12 heads, 110M parameters
- **Student:** 6 layers, $d=768$, 12 heads, FFN 3072, 66M parameters; no token-type embeddings, no pooler
- **Initialisation:** from the teacher, one layer out of two
- **Data:** the same corpus as BERT: English Wikipedia and the Toronto Book Corpus
- **Recipe:** RoBERTa-style: very large batches (up to 4K examples) via gradient accumulation, dynamic masking, no next-sentence prediction
- **Compute:** 8 × 16GB V100 GPUs for about 90 hours (the paper contrasts RoBERTa: 1 day on 1024 × 32GB V100s)
- **GLUE (dev):** macro-score 77.0 vs 79.5 for BERT-base and 68.7 for ELMo: 97% of BERT's score
- **Speed:** CPU, batch size 1, one pass over STS-B: 410 s vs 668 s for BERT-base (60% faster) and 895 s for ELMo
- **Downstream:** IMDb accuracy 92.82 vs 93.46; SQuAD v1.1 EM/F1 77.7/85.8 vs 81.2/88.5, and 79.1/86.9 with a second distillation step during fine-tuning
- **On device:** iPhone 7 Plus question answering: 71% faster than BERT-base (excluding tokenization), 207 MB of weights

> Figure: One masked position, "the cat sat on the [MASK] .", over 8 candidate words with made-up teacher and student logits. The bars are the teacher's and student's probabilities at temperature T. At the default T = 2 with an untrained student, the teacher puts 38.2% on mat (65.2% at T = 1) and its entropy is 1.732 nats (ln 8 = 2.079 is the maximum). KL(t‖s) = 0.271, so L_ce = T²·KL = 1.083, L_mlm = 1.743 and L_cos = 0.464 (a 57.6° angle between the 16-dimensional hidden states). The weighted total 5·1.083 + 2·1.743 + 0.464 is 9.366. Raising T shrinks the raw KL fast (0.733, 0.271, 0.081, 0.022 at T = 1, 2, 4, 8) while T²·KL stays in the same range (0.733, 1.083, 1.287, 1.385). The teacher's ratio p(floor)/p(car) falls from 164.0 at T = 1 to 12.8 at T = 2 and 1.9 at T = 8, so the tail words carry more weight. At 50% progress, L_ce = 0.250, L_mlm = 0.867, L_cos = 0.114 (27.7°) and the total is 3.098. At 100% the student equals the teacher: L_ce and L_cos are 0, but L_mlm stays at 0.427, the teacher's own error on the gold word, for a total of 0.854.

## Worked example: the three losses at one position

Take a 3-word vocabulary with the gold word first. Teacher logits $z^t=(3,1,0)$, student logits $z^s=(2,1.5,0)$, temperature $T=2$. Hidden states (3-D for the example) $h^t=(1,0,1)$ and $h^s=(1,1,0)$. Weights 5, 2 and 1.

Softened and unsoftened distributions (computed in numpy)

| quantity | word 1 (gold) | word 2 | word 3 |
| --- | --- | --- | --- |
| teacher t = softmax(z_t / 2) | 0.6285 | 0.2312 | 0.1402 |
| student s = softmax(z_s / 2) | 0.4658 | 0.3628 | 0.1714 |
| student softmax(z_s), T = 1 | 0.5741 | 0.3482 | 0.0777 |
| gradient T·(s − t) | −0.3254 | +0.2631 | +0.0623 |

**Assembling the loss**

1. Soft cross-entropy: $-\sum_i t_i\log s_i=0.9620$. Teacher entropy $H(t)=0.9060$, so $\mathrm{KL}(t\|s)=0.0560$ and $T^2\cdot\mathrm{KL}=0.2241$.
2. Masked LM at $T=1$: $L_{mlm}=-\log 0.5741=0.5550$.
3. Cosine: $h^s\cdot h^t=1$ and $\|h^s\|\|h^t\|=2$, so $\cos=0.5$ and $L_{cos}=0.5$.
4. Total: $5(0.2241)+2(0.5550)+1(0.5)=1.1206+1.1100+0.5=2.7302$.
5. The gradient row shows the soft term pushing word 1 up and word 2 down. The student over-rates word 2 (0.3628 vs 0.2312) even though it ranks word 1 first. A hard label would only say "word 1"; the soft target also says *by how much* each wrong word should lose.

The parameter count follows from the architecture. One BERT-base layer has $4(768^2+768)$ attention weights, $2\cdot768\cdot3072+3072+768$ FFN weights and two LayerNorms: 7,087,872 parameters. BERT-base is $23.8\text{M}$ of embeddings, 12 layers and a 0.6M pooler, about 109.5M in total. DistilBERT keeps the token and position embeddings, drops the segment embeddings and the pooler, and has 6 layers: 66.4M. That is 39.4% fewer parameters, the paper's "40% smaller".

## What the ablation shows

The paper's ablation reports the change in GLUE macro-score when one part of the recipe is removed. Dropping the distillation loss $L_{ce}$ (keeping $L_{mlm}$ and $L_{cos}$) cost 2.96 points. Dropping the cosine loss cost 1.46. Dropping the MLM loss cost only 0.31. The largest drop, 3.69 points, came from keeping all three losses but initialising the student **randomly** instead of from the teacher.

So the student's head start matters more than any one loss term. The two distillation signals carry most of the benefit, and the hard MLM label adds little once the teacher's soft targets are there. The soft targets already contain the gold word, usually with high probability.

```python
import torch, torch.nn.functional as F
from transformers import (BertForMaskedLM, BertTokenizerFast, DistilBertConfig,
                          DistilBertForMaskedLM, DataCollatorForLanguageModeling)

tok = BertTokenizerFast.from_pretrained("bert-base-uncased")
teacher = BertForMaskedLM.from_pretrained("bert-base-uncased").eval()
student = DistilBertForMaskedLM(DistilBertConfig(vocab_size=tok.vocab_size, n_layers=6, dim=768,
                                                 n_heads=12, hidden_dim=3072))

# ---- initialise the student from the teacher, one layer out of two ----
tb, sb = teacher.bert, student.distilbert
sb.embeddings.word_embeddings.load_state_dict(tb.embeddings.word_embeddings.state_dict())
sb.embeddings.position_embeddings.load_state_dict(tb.embeddings.position_embeddings.state_dict())
sb.embeddings.LayerNorm.load_state_dict(tb.embeddings.LayerNorm.state_dict())  # no token-type embeddings
for j, i in enumerate([0, 2, 4, 7, 9, 11]):        # the layers the released extraction script keeps
    t, s = tb.encoder.layer[i], sb.transformer.layer[j]
    for dst, src in [(s.attention.q_lin, t.attention.self.query), (s.attention.k_lin, t.attention.self.key),
                     (s.attention.v_lin, t.attention.self.value), (s.attention.out_lin, t.attention.output.dense),
                     (s.sa_layer_norm, t.attention.output.LayerNorm), (s.ffn.lin1, t.intermediate.dense),
                     (s.ffn.lin2, t.output.dense), (s.output_layer_norm, t.output.LayerNorm)]:
        dst.load_state_dict(src.state_dict())
head = teacher.cls.predictions
student.vocab_transform.load_state_dict(head.transform.dense.state_dict())
student.vocab_layer_norm.load_state_dict(head.transform.LayerNorm.state_dict())
student.vocab_projector.bias.data.copy_(head.bias.data)   # projector weight is tied to the word embeddings

# ---- one training step on the triple loss ----
T, a_ce, a_mlm, a_cos = 2.0, 5.0, 2.0, 1.0
collate = DataCollatorForLanguageModeling(tok, mlm_probability=0.15)   # dynamic masking, no NSP
batch = collate([tok(t) for t in ["the cat sat on the mat .", "distillation makes models smaller ."]])
opt = torch.optim.AdamW(student.parameters(), lr=5e-4)

with torch.no_grad():
    t_out = teacher(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"],
                    output_hidden_states=True)
s_out = student(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"],
                labels=batch["labels"], output_hidden_states=True)

keep = batch["attention_mask"].bool()                      # every non-padding position
s_logits, t_logits = s_out.logits[keep], t_out.logits[keep]  # (n_tokens, 30522)
loss_ce = F.kl_div(F.log_softmax(s_logits / T, -1), F.softmax(t_logits / T, -1),
                   reduction="batchmean") * T ** 2        # T²·KL(t || s)
loss_mlm = s_out.loss                                      # cross-entropy on the masked positions only
h_s, h_t = s_out.hidden_states[-1][keep], t_out.hidden_states[-1][keep]   # (n_tokens, 768)
loss_cos = F.cosine_embedding_loss(h_s, h_t, torch.ones(h_s.size(0)))     # mean of 1 − cos

loss = a_ce * loss_ce + a_mlm * loss_mlm + a_cos * loss_cos
opt.zero_grad(); loss.backward(); opt.step()
print(sum(p.numel() for p in student.parameters()) / 1e6)  # about 66M (tied projector counted once)
```

DistilBERT-style distillation step with Hugging Face transformers: copy every other teacher layer into a 6-layer DistilBERT, then train on the triple loss (written for transformers 4.x; not run here)

> **Forgetting the T² factor:** If you sweep $T$ without the $T^2$ factor, the soft term's gradient shrinks roughly like $1/T^2$. The hard MLM term then dominates, and a "higher temperature" run quietly becomes a no-distillation run. In the figure, the raw KL drops from 0.733 to 0.022 between $T=1$ and $T=8$ while $T^2\cdot$KL stays between 0.733 and 1.385. Also remember to divide **both** logit sets by $T$: softening only the teacher's distribution mismatches the student's.

> **Depth, not width, sets the latency:** DistilBERT halves the layers and keeps $d=768$. Each layer is a sequential step, while the matrix products within a layer run in parallel on wide hardware. Halving depth therefore cuts latency almost in proportion: 410 s vs 668 s on the paper's CPU benchmark. Halving the width would remove a similar number of parameters but save much less time. It would also stop the student from copying the teacher's weights directly.

> **Task-agnostic, then optionally task-specific:** DistilBERT is distilled once, on raw text, and fine-tuned like BERT. The paper also tried a second distillation step during SQuAD fine-tuning, with a fine-tuned BERT as teacher. This raised EM/F1 from 77.7/85.8 to 79.1/86.9, 1.6 F1 behind BERT-base (88.5). TinyBERT, MobileBERT and MiniLM later distilled attention maps and intermediate layers as well. See **Knowledge distillation** for the general framework.

**Strengths**

- 40% fewer parameters and 60% faster inference for 97% of BERT-base's GLUE score
- Drop-in replacement: same tokenizer, width and fine-tuning recipe as BERT
- Cheap to produce: about 90 hours on 8 V100s, using a teacher that already exists
- General-purpose: one distillation run serves every downstream task

**Limitations**

- Gaps remain on harder tasks: 2.7 F1 behind BERT-base on SQuAD v1.1 without task-specific distillation
- Needs the teacher in the loop: every training step pays a forward pass of the 12-layer model
- The student is tied to the teacher's width and tokenizer by the layer-copy initialisation
- Matches only final outputs and last hidden states; later methods distil intermediate layers and attention too

- **Low-latency classification:** sentiment, intent and toxicity classifiers in production, where 6 layers instead of 12 roughly halves serving cost
- **On-device NLP:** question answering and classification on phones; the paper's iPhone 7 Plus demo ran 71% faster than BERT-base
- **Embedding and retrieval baselines:** a cheap encoder for sentence embeddings and re-ranking, often fine-tuned further

**Question:** Why does the distillation loss multiply KL(t‖s) by T²?

- A. To make the loss non-negative
- B. Because the gradient of the softened KL with respect to the student logits shrinks roughly like 1/T², so T² keeps its size comparable to the hard MLM loss
- C. To make the teacher's distribution sharper
- D. Because the cosine loss is measured in squared units

**Answer:** B. Because the gradient of the softened KL with respect to the student logits shrinks roughly like 1/T², so T² keeps its size comparable to the hard MLM loss — $\partial\,\mathrm{KL}/\partial z^s=(s-t)/T$, and $s-t$ itself shrinks with $T$. Multiplying by $T^2$ (Hinton et al. 2015) keeps the relative weight of the soft and hard terms stable when the temperature changes.

**Question:** In the DistilBERT ablation, which change hurt the GLUE macro-score the most?

- A. Dropping the masked-LM loss
- B. Dropping the cosine loss
- C. Initialising the student randomly instead of from every other teacher layer
- D. Removing the token-type embeddings

**Answer:** C. Initialising the student randomly instead of from every other teacher layer — Random initialisation cost 3.69 points, more than any single loss change. Dropping the distillation loss cost 2.96, dropping the cosine loss 1.46, and dropping the MLM loss only 0.31.

**Key points**

- DistilBERT: 6 layers, $d=768$, 12 heads, 66M parameters, distilled from BERT-base during pre-training on Wikipedia and BookCorpus.
- Triple loss: $\alpha_{ce}L_{ce}+\alpha_{mlm}L_{mlm}+\alpha_{cos}L_{cos}$, with soft targets $\operatorname{softmax}(z^t/T)$ for teacher and student, BERT's MLM loss, and $1-\cos(h^s,h^t)$ on the last hidden states.
- Temperature exposes the teacher's ranking of wrong answers; the $T^2$ factor keeps the soft term's gradient from vanishing.
- Initialisation from one teacher layer out of two is the most important ingredient (−3.69 GLUE points without it).
- Result: 40% smaller, 60% faster, 97% of BERT-base's GLUE macro-score (77.0 vs 79.5).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
