DistilBERT

DistilBERT (Sanh et al., 2019) is a 6-layer BERT distilled during pre-training with a triple loss: soft-target cross-entropy at temperature T, masked LM and a hidden-state cosine loss. Initialised from every other teacher layer, it is 40% smaller, 60% faster and keeps 97% of BERT-base's GLUE score.

14 min read1067 wordscore0 viewsupdated 2026-10-13

Definition

DistilBERT is the compressed BERT of Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf (Hugging Face), "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter" (arXiv, October 2019; NeurIPS 2019 EMC² workshop). A 6-layer student with BERT-base's width is trained to imitate the 12-layer BERT-base teacher during pre-training, not just on one downstream task. The objective is a sum of three terms: a distillation cross-entropy against the teacher's temperature-softened output distribution, the usual masked-language-model loss, and a cosine loss that aligns the student's and teacher's final hidden states. The student starts from the teacher's weights, taking one layer out of two.

By 2019 pre-trained language models were growing fast: BERT-large had 340M parameters, and newer models went well beyond that. Running them on a CPU, a phone or under a tight latency budget was costly. Knowledge distillation (Buciluǎ et al. 2006; Hinton, Vinyals and Dean 2015) offered a fix: train a small student to reproduce the outputs of a large teacher. Earlier work on BERT had distilled task-specific models, for example into BiLSTMs for one classification task. DistilBERT applied distillation to the general-purpose model instead. The student is distilled once, on the pre-training corpus, and can then be fine-tuned on any task like BERT itself.

The architecture follows from one observation in the paper. For a fixed parameter budget, the hidden size has a smaller effect on computation efficiency than the number of layers, because the matrix multiplications along the last dimension are highly optimised. So the authors kept d=768d=768 and 12 heads and halved the depth. They also removed the token-type (segment) embeddings and the pooler, which only served next-sentence prediction and the [CLS] classifier.

Architecture and initialisation

MLM head → logits z_tlayer 1layer 2layer 3layer 4layer 5layer 6layer 7layer 8layer 9layer 10layer 11layer 12token + position + segmentteacher: BERT-base, frozen12 × (d=768, 12 heads), 110MMLM head → logits z_slayer 1 ← teacher 1layer 2 ← teacher 3layer 3 ← teacher 5layer 4 ← teacher 8layer 5 ← teacher 10layer 6 ← teacher 12token + position (copied)student: DistilBERT6 × (d=768, 12 heads), 66Minit: one layerout of twono token-type emb.no poolerL_ce distillationsoftmax(z_t/T) vssoftmax(z_s/T)L_mlm masked LM−log softmax(z_s)[gold]L_cos hidden states1 − cos(h_s, h_t)total lossα_ce·L_ce + α_mlm·L_mlm+ α_cos·L_cos (5, 2, 1)15% of tokens masked,dynamic masking, no NSP,batches up to 4K sequencesT = temperature, both sidesz_tz_sh_th_s
fig.Teacher and student side by side. Each student layer is drawn level with the teacher layer whose weights it starts from: one layer out of two, with the released extraction script copying teacher layers 1, 3, 5, 8, 10 and 12. Both models produce logits over the 30,522-token vocabulary at every position. The teacher's logits z_t and the student's z_s are softened by the same temperature T for the distillation loss. The student's unsoftened logits feed the masked-LM loss, and the two last hidden states feed the cosine loss. The teacher is frozen; only the student is trained.

The student is a BERT encoder with the same embeddings, attention blocks and feed-forward blocks, but 6 layers instead of 12. Because the shapes match, its weights can be copied directly from the teacher. The paper initialises it by "taking one layer out of two", which gives the student a starting point that already computes useful contextual features. The token and position embeddings and the MLM head are copied too. The ablation below shows this initialisation is the single most important ingredient.

The triple loss

ti=exp⁡(zit/T)∑jexp⁡(zjt/T),si=exp⁡(zis/T)∑jexp⁡(zjs/T),Lce=−∑itilog⁡sit_i=\frac{\exp(z^{t}_i/T)}{\sum_j\exp(z^{t}_j/T)},\qquad s_i=\frac{\exp(z^{s}_i/T)}{\sum_j\exp(z^{s}_j/T)},\qquad L_{ce}=-\sum_i t_i\log s_i
zt, zsz^t,\ z^s
teacher and student logits over the vocabulary at one position
TT
softmax temperature, applied to both models in training; T=1T=1 at inference
tit_i
soft target: the teacher's probability for word i
LceL_{ce}
distillation cross-entropy; it equals KL(t ∥ s)+H(t)\mathrm{KL}(t\,\|\,s)+H(t), and H(t)H(t) does not depend on the student

The soft targets carry what Hinton called dark knowledge. For "the cat sat on the [MASK]", a teacher that puts 65% on mat still ranks floor far above car. A one-hot label throws that ranking away. Raising TT flattens the distribution so that the small probabilities, and therefore the ranking among wrong answers, contribute more to the loss.

∂ KL(t∥s)∂zis=1T (si−ti)⟹∂ T2KL(t∥s)∂zis=T (si−ti)\frac{\partial\,\mathrm{KL}(t\|s)}{\partial z^s_i}=\frac{1}{T}\,(s_i-t_i)\qquad\Longrightarrow\qquad \frac{\partial\,T^2\mathrm{KL}(t\|s)}{\partial z^s_i}=T\,(s_i-t_i)
1/T1/T
the chain rule through z/Tz/T; and since s−ts-t itself shrinks roughly like 1/T1/T for large TT, the raw gradient falls like 1/T21/T^2
T2T^2
Hinton et al.'s correction, used in the Hugging Face distiller, so the soft term keeps its weight relative to the hard MLM term when TT changes
Lmlm=−log⁡softmax⁡(zs)y,Lcos=1−cos⁡(hs,ht),L=αceLce+αmlmLmlm+αcosLcosL_{mlm}=-\log\operatorname{softmax}(z^s)_{y},\qquad L_{cos}=1-\cos\big(h^s,h^t\big),\qquad L=\alpha_{ce}L_{ce}+\alpha_{mlm}L_{mlm}+\alpha_{cos}L_{cos}
yy
the original token at a masked position (BERT's MLM, at T = 1)
hs, hth^s,\ h^t
last-layer hidden states of student and teacher at the same position, both 768-dimensional
LcosL_{cos}
aligns the directions of the hidden-state vectors, not their lengths
α\alpha
the paper gives no values; the example command in the released training code uses αce=5\alpha_{ce}=5, αmlm=2\alpha_{mlm}=2, αcos=1\alpha_{cos}=1

Each term does a different job. LceL_{ce} transfers the teacher's full output distribution. LmlmL_{mlm} keeps the student anchored to the real data, which matters where the teacher is wrong. LcosL_{cos} adds a signal at the representation level: the hidden states, not just the predictions, should point the same way. The cosine form leaves the norm free, so the student does not have to match the teacher's scale exactly.

DistilBERT in the paper

Teacher
BERT-base: 12 layers, d=768d=768, 12 heads, 110M parameters
Student
6 layers, d=768d=768, 12 heads, FFN 3072, 66M parameters; no token-type embeddings, no pooler
Initialisation
from the teacher, one layer out of two
Data
the same corpus as BERT: English Wikipedia and the Toronto Book Corpus
Recipe
RoBERTa-style: very large batches (up to 4K examples) via gradient accumulation, dynamic masking, no next-sentence prediction
Compute
8 × 16GB V100 GPUs for about 90 hours (the paper contrasts RoBERTa: 1 day on 1024 × 32GB V100s)
GLUE (dev)
macro-score 77.0 vs 79.5 for BERT-base and 68.7 for ELMo: 97% of BERT's score
Speed
CPU, batch size 1, one pass over STS-B: 410 s vs 668 s for BERT-base (60% faster) and 895 s for ELMo
Downstream
IMDb accuracy 92.82 vs 93.46; SQuAD v1.1 EM/F1 77.7/85.8 vs 81.2/88.5, and 79.1/86.9 with a second distillation step during fine-tuning
On device
iPhone 7 Plus question answering: 71% faster than BERT-base (excluding tokenization), 207 MB of weights
“the cat sat on the [MASK] .” · softmax(z / T) at T = 2.0 · student progress 0%■ teacher t = softmax(z_t/T) · ■ student s = softmax(z_s/T) · ▬ teacher at T = 10.00.20.40.6mat38.215.7floor19.019.2sofa14.17.1bed11.514.2roof6.317.4chair5.46.4table4.010.5car1.59.5t%s%teacher entropy H(t)1.732 natsKL(t ‖ s) at T0.271L_ce = T²·KL1.083L_mlm = −log s(mat)1.743L_cos = 1 − cos(h_s, h_t)0.4645·L_ce + 2·L_mlm + 1·L_cos9.366h_th_sangle 57.6° in d = 16unscaled KL shrinks like 1/T²; T² keeps L_ce on the scale of L_mlm∂L_ce/∂z_s = T·(s − t), norm 0.563teacher p(floor)/p(car): 12.8 at T, 164.0 at T = 1soft-target CE −Σ t·log s = 2.003 = H(t) + KLraise progress to move z_s and h_s toward the teacherweights α_ce = 5, α_mlm = 2, α_cos = 1 as in the released training script; logits are made up
reference line
fig.One masked position, "the cat sat on the [MASK] .", over 8 candidate words with made-up teacher and student logits. The bars are the teacher's and student's probabilities at temperature T. At the default T = 2 with an untrained student, the teacher puts 38.2% on mat (65.2% at T = 1) and its entropy is 1.732 nats (ln 8 = 2.079 is the maximum). KL(t‖s) = 0.271, so L_ce = T²·KL = 1.083, L_mlm = 1.743 and L_cos = 0.464 (a 57.6° angle between the 16-dimensional hidden states). The weighted total 5·1.083 + 2·1.743 + 0.464 is 9.366. Raising T shrinks the raw KL fast (0.733, 0.271, 0.081, 0.022 at T = 1, 2, 4, 8) while T²·KL stays in the same range (0.733, 1.083, 1.287, 1.385). The teacher's ratio p(floor)/p(car) falls from 164.0 at T = 1 to 12.8 at T = 2 and 1.9 at T = 8, so the tail words carry more weight. At 50% progress, L_ce = 0.250, L_mlm = 0.867, L_cos = 0.114 (27.7°) and the total is 3.098. At 100% the student equals the teacher: L_ce and L_cos are 0, but L_mlm stays at 0.427, the teacher's own error on the gold word, for a total of 0.854.

Worked example: the three losses at one position

Take a 3-word vocabulary with the gold word first. Teacher logits zt=(3,1,0)z^t=(3,1,0), student logits zs=(2,1.5,0)z^s=(2,1.5,0), temperature T=2T=2. Hidden states (3-D for the example) ht=(1,0,1)h^t=(1,0,1) and hs=(1,1,0)h^s=(1,1,0). Weights 5, 2 and 1.

quantityword 1 (gold)word 2word 3
teacher t = softmax(z_t / 2)0.62850.23120.1402
student s = softmax(z_s / 2)0.46580.36280.1714
student softmax(z_s), T = 10.57410.34820.0777
gradient T·(s − t)−0.3254+0.2631+0.0623
Softened and unsoftened distributions (computed in numpy)

Assembling the loss

  1. 1Soft cross-entropy: −∑itilog⁡si=0.9620-\sum_i t_i\log s_i=0.9620. Teacher entropy H(t)=0.9060H(t)=0.9060, so KL(t∥s)=0.0560\mathrm{KL}(t\|s)=0.0560 and T2⋅KL=0.2241T^2\cdot\mathrm{KL}=0.2241.
  2. 2Masked LM at T=1T=1: Lmlm=−log⁡0.5741=0.5550L_{mlm}=-\log 0.5741=0.5550.
  3. 3Cosine: hs⋅ht=1h^s\cdot h^t=1 and ∥hs∥∥ht∥=2\|h^s\|\|h^t\|=2, so cos⁡=0.5\cos=0.5 and Lcos=0.5L_{cos}=0.5.
  4. 4Total: 5(0.2241)+2(0.5550)+1(0.5)=1.1206+1.1100+0.5=2.73025(0.2241)+2(0.5550)+1(0.5)=1.1206+1.1100+0.5=2.7302.
  5. 5The gradient row shows the soft term pushing word 1 up and word 2 down. The student over-rates word 2 (0.3628 vs 0.2312) even though it ranks word 1 first. A hard label would only say "word 1"; the soft target also says by how much each wrong word should lose.

The parameter count follows from the architecture. One BERT-base layer has 4(7682+768)4(768^2+768) attention weights, 2⋅768⋅3072+3072+7682\cdot768\cdot3072+3072+768 FFN weights and two LayerNorms: 7,087,872 parameters. BERT-base is 23.8M23.8\text{M} of embeddings, 12 layers and a 0.6M pooler, about 109.5M in total. DistilBERT keeps the token and position embeddings, drops the segment embeddings and the pooler, and has 6 layers: 66.4M. That is 39.4% fewer parameters, the paper's "40% smaller".

What the ablation shows

The paper's ablation reports the change in GLUE macro-score when one part of the recipe is removed. Dropping the distillation loss LceL_{ce} (keeping LmlmL_{mlm} and LcosL_{cos}) cost 2.96 points. Dropping the cosine loss cost 1.46. Dropping the MLM loss cost only 0.31. The largest drop, 3.69 points, came from keeping all three losses but initialising the student randomly instead of from the teacher.

So the student's head start matters more than any one loss term. The two distillation signals carry most of the benefit, and the hard MLM label adds little once the teacher's soft targets are there. The soft targets already contain the gold word, usually with high probability.

python
1import torch, torch.nn.functional as F
2from transformers import (BertForMaskedLM, BertTokenizerFast, DistilBertConfig,
3 DistilBertForMaskedLM, DataCollatorForLanguageModeling)
4
5tok = BertTokenizerFast.from_pretrained("bert-base-uncased")
6teacher = BertForMaskedLM.from_pretrained("bert-base-uncased").eval()
7student = DistilBertForMaskedLM(DistilBertConfig(vocab_size=tok.vocab_size, n_layers=6, dim=768,
8 n_heads=12, hidden_dim=3072))
9
10# ---- initialise the student from the teacher, one layer out of two ----
11tb, sb = teacher.bert, student.distilbert
12sb.embeddings.word_embeddings.load_state_dict(tb.embeddings.word_embeddings.state_dict())
13sb.embeddings.position_embeddings.load_state_dict(tb.embeddings.position_embeddings.state_dict())
14sb.embeddings.LayerNorm.load_state_dict(tb.embeddings.LayerNorm.state_dict()) # no token-type embeddings
15for j, i in enumerate([0, 2, 4, 7, 9, 11]): # the layers the released extraction script keeps
16 t, s = tb.encoder.layer[i], sb.transformer.layer[j]
17 for dst, src in [(s.attention.q_lin, t.attention.self.query), (s.attention.k_lin, t.attention.self.key),
18 (s.attention.v_lin, t.attention.self.value), (s.attention.out_lin, t.attention.output.dense),
19 (s.sa_layer_norm, t.attention.output.LayerNorm), (s.ffn.lin1, t.intermediate.dense),
20 (s.ffn.lin2, t.output.dense), (s.output_layer_norm, t.output.LayerNorm)]:
21 dst.load_state_dict(src.state_dict())
22head = teacher.cls.predictions
23student.vocab_transform.load_state_dict(head.transform.dense.state_dict())
24student.vocab_layer_norm.load_state_dict(head.transform.LayerNorm.state_dict())
25student.vocab_projector.bias.data.copy_(head.bias.data) # projector weight is tied to the word embeddings
26
27# ---- one training step on the triple loss ----
28T, a_ce, a_mlm, a_cos = 2.0, 5.0, 2.0, 1.0
29collate = DataCollatorForLanguageModeling(tok, mlm_probability=0.15) # dynamic masking, no NSP
30batch = collate([tok(t) for t in ["the cat sat on the mat .", "distillation makes models smaller ."]])
31opt = torch.optim.AdamW(student.parameters(), lr=5e-4)
32
33with torch.no_grad():
34 t_out = teacher(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"],
35 output_hidden_states=True)
36s_out = student(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"],
37 labels=batch["labels"], output_hidden_states=True)
38
39keep = batch["attention_mask"].bool() # every non-padding position
40s_logits, t_logits = s_out.logits[keep], t_out.logits[keep] # (n_tokens, 30522)
41loss_ce = F.kl_div(F.log_softmax(s_logits / T, -1), F.softmax(t_logits / T, -1),
42 reduction="batchmean") * T ** 2 # T²·KL(t || s)
43loss_mlm = s_out.loss # cross-entropy on the masked positions only
44h_s, h_t = s_out.hidden_states[-1][keep], t_out.hidden_states[-1][keep] # (n_tokens, 768)
45loss_cos = F.cosine_embedding_loss(h_s, h_t, torch.ones(h_s.size(0))) # mean of 1 − cos
46
47loss = a_ce * loss_ce + a_mlm * loss_mlm + a_cos * loss_cos
48opt.zero_grad(); loss.backward(); opt.step()
49print(sum(p.numel() for p in student.parameters()) / 1e6) # about 66M (tied projector counted once)
DistilBERT-style distillation step with Hugging Face transformers: copy every other teacher layer into a 6-layer DistilBERT, then train on the triple loss (written for transformers 4.x; not run here)

Use when

  • +40% fewer parameters and 60% faster inference for 97% of BERT-base's GLUE score
  • +Drop-in replacement: same tokenizer, width and fine-tuning recipe as BERT
  • +Cheap to produce: about 90 hours on 8 V100s, using a teacher that already exists
  • +General-purpose: one distillation run serves every downstream task

Avoid when

  • −Gaps remain on harder tasks: 2.7 F1 behind BERT-base on SQuAD v1.1 without task-specific distillation
  • −Needs the teacher in the loop: every training step pays a forward pass of the 12-layer model
  • −The student is tied to the teacher's width and tokenizer by the layer-copy initialisation
  • −Matches only final outputs and last hidden states; later methods distil intermediate layers and attention too

Where it's used

Low-latency classification

sentiment, intent and toxicity classifiers in production, where 6 layers instead of 12 roughly halves serving cost

On-device NLP

question answering and classification on phones; the paper's iPhone 7 Plus demo ran 71% faster than BERT-base

Embedding and retrieval baselines

a cheap encoder for sentence embeddings and re-ranking, often fine-tuned further

Check yourself

Why does the distillation loss multiply KL(t‖s) by T²?

Check yourself

In the DistilBERT ablation, which change hurt the GLUE macro-score the most?

Remember

  • ✓DistilBERT: 6 layers, d=768d=768, 12 heads, 66M parameters, distilled from BERT-base during pre-training on Wikipedia and BookCorpus.
  • ✓Triple loss: αceLce+αmlmLmlm+αcosLcos\alpha_{ce}L_{ce}+\alpha_{mlm}L_{mlm}+\alpha_{cos}L_{cos}, with soft targets softmax⁡(zt/T)\operatorname{softmax}(z^t/T) for teacher and student, BERT's MLM loss, and 1−cos⁡(hs,ht)1-\cos(h^s,h^t) on the last hidden states.
  • ✓Temperature exposes the teacher's ranking of wrong answers; the T2T^2 factor keeps the soft term's gradient from vanishing.
  • ✓Initialisation from one teacher layer out of two is the most important ingredient (−3.69 GLUE points without it).
  • ✓Result: 40% smaller, 60% faster, 97% of BERT-base's GLUE macro-score (77.0 vs 79.5).
← back to Deep Learning roadmap