Definition
DistilBERT is the compressed BERT of Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf (Hugging Face), "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter" (arXiv, October 2019; NeurIPS 2019 EMC² workshop). A 6-layer student with BERT-base's width is trained to imitate the 12-layer BERT-base teacher during pre-training, not just on one downstream task. The objective is a sum of three terms: a distillation cross-entropy against the teacher's temperature-softened output distribution, the usual masked-language-model loss, and a cosine loss that aligns the student's and teacher's final hidden states. The student starts from the teacher's weights, taking one layer out of two.
By 2019 pre-trained language models were growing fast: BERT-large had 340M parameters, and newer models went well beyond that. Running them on a CPU, a phone or under a tight latency budget was costly. Knowledge distillation (Buciluǎ et al. 2006; Hinton, Vinyals and Dean 2015) offered a fix: train a small student to reproduce the outputs of a large teacher. Earlier work on BERT had distilled task-specific models, for example into BiLSTMs for one classification task. DistilBERT applied distillation to the general-purpose model instead. The student is distilled once, on the pre-training corpus, and can then be fine-tuned on any task like BERT itself.
The architecture follows from one observation in the paper. For a fixed parameter budget, the hidden size has a smaller effect on computation efficiency than the number of layers, because the matrix multiplications along the last dimension are highly optimised. So the authors kept and 12 heads and halved the depth. They also removed the token-type (segment) embeddings and the pooler, which only served next-sentence prediction and the [CLS] classifier.
Architecture and initialisation
The student is a BERT encoder with the same embeddings, attention blocks and feed-forward blocks, but 6 layers instead of 12. Because the shapes match, its weights can be copied directly from the teacher. The paper initialises it by "taking one layer out of two", which gives the student a starting point that already computes useful contextual features. The token and position embeddings and the MLM head are copied too. The ablation below shows this initialisation is the single most important ingredient.
The triple loss
- teacher and student logits over the vocabulary at one position
- softmax temperature, applied to both models in training; at inference
- soft target: the teacher's probability for word i
- distillation cross-entropy; it equals , and does not depend on the student
The soft targets carry what Hinton called dark knowledge. For "the cat sat on the [MASK]", a teacher that puts 65% on mat still ranks floor far above car. A one-hot label throws that ranking away. Raising flattens the distribution so that the small probabilities, and therefore the ranking among wrong answers, contribute more to the loss.
- the chain rule through ; and since itself shrinks roughly like for large , the raw gradient falls like
- Hinton et al.'s correction, used in the Hugging Face distiller, so the soft term keeps its weight relative to the hard MLM term when changes
- the original token at a masked position (BERT's MLM, at T = 1)
- last-layer hidden states of student and teacher at the same position, both 768-dimensional
- aligns the directions of the hidden-state vectors, not their lengths
- the paper gives no values; the example command in the released training code uses , ,
Each term does a different job. transfers the teacher's full output distribution. keeps the student anchored to the real data, which matters where the teacher is wrong. adds a signal at the representation level: the hidden states, not just the predictions, should point the same way. The cosine form leaves the norm free, so the student does not have to match the teacher's scale exactly.
DistilBERT in the paper
- Teacher
- BERT-base: 12 layers, , 12 heads, 110M parameters
- Student
- 6 layers, , 12 heads, FFN 3072, 66M parameters; no token-type embeddings, no pooler
- Initialisation
- from the teacher, one layer out of two
- Data
- the same corpus as BERT: English Wikipedia and the Toronto Book Corpus
- Recipe
- RoBERTa-style: very large batches (up to 4K examples) via gradient accumulation, dynamic masking, no next-sentence prediction
- Compute
- 8 × 16GB V100 GPUs for about 90 hours (the paper contrasts RoBERTa: 1 day on 1024 × 32GB V100s)
- GLUE (dev)
- macro-score 77.0 vs 79.5 for BERT-base and 68.7 for ELMo: 97% of BERT's score
- Speed
- CPU, batch size 1, one pass over STS-B: 410 s vs 668 s for BERT-base (60% faster) and 895 s for ELMo
- Downstream
- IMDb accuracy 92.82 vs 93.46; SQuAD v1.1 EM/F1 77.7/85.8 vs 81.2/88.5, and 79.1/86.9 with a second distillation step during fine-tuning
- On device
- iPhone 7 Plus question answering: 71% faster than BERT-base (excluding tokenization), 207 MB of weights
Worked example: the three losses at one position
Take a 3-word vocabulary with the gold word first. Teacher logits , student logits , temperature . Hidden states (3-D for the example) and . Weights 5, 2 and 1.
| quantity | word 1 (gold) | word 2 | word 3 |
|---|---|---|---|
| teacher t = softmax(z_t / 2) | 0.6285 | 0.2312 | 0.1402 |
| student s = softmax(z_s / 2) | 0.4658 | 0.3628 | 0.1714 |
| student softmax(z_s), T = 1 | 0.5741 | 0.3482 | 0.0777 |
| gradient T·(s − t) | −0.3254 | +0.2631 | +0.0623 |
Assembling the loss
- 1Soft cross-entropy: . Teacher entropy , so and .
- 2Masked LM at : .
- 3Cosine: and , so and .
- 4Total: .
- 5The gradient row shows the soft term pushing word 1 up and word 2 down. The student over-rates word 2 (0.3628 vs 0.2312) even though it ranks word 1 first. A hard label would only say "word 1"; the soft target also says by how much each wrong word should lose.
The parameter count follows from the architecture. One BERT-base layer has attention weights, FFN weights and two LayerNorms: 7,087,872 parameters. BERT-base is of embeddings, 12 layers and a 0.6M pooler, about 109.5M in total. DistilBERT keeps the token and position embeddings, drops the segment embeddings and the pooler, and has 6 layers: 66.4M. That is 39.4% fewer parameters, the paper's "40% smaller".
What the ablation shows
The paper's ablation reports the change in GLUE macro-score when one part of the recipe is removed. Dropping the distillation loss (keeping and ) cost 2.96 points. Dropping the cosine loss cost 1.46. Dropping the MLM loss cost only 0.31. The largest drop, 3.69 points, came from keeping all three losses but initialising the student randomly instead of from the teacher.
So the student's head start matters more than any one loss term. The two distillation signals carry most of the benefit, and the hard MLM label adds little once the teacher's soft targets are there. The soft targets already contain the gold word, usually with high probability.
1import torch, torch.nn.functional as F2from transformers import (BertForMaskedLM, BertTokenizerFast, DistilBertConfig,3 DistilBertForMaskedLM, DataCollatorForLanguageModeling)45tok = BertTokenizerFast.from_pretrained("bert-base-uncased")6teacher = BertForMaskedLM.from_pretrained("bert-base-uncased").eval()7student = DistilBertForMaskedLM(DistilBertConfig(vocab_size=tok.vocab_size, n_layers=6, dim=768,8 n_heads=12, hidden_dim=3072))910# ---- initialise the student from the teacher, one layer out of two ----11tb, sb = teacher.bert, student.distilbert12sb.embeddings.word_embeddings.load_state_dict(tb.embeddings.word_embeddings.state_dict())13sb.embeddings.position_embeddings.load_state_dict(tb.embeddings.position_embeddings.state_dict())14sb.embeddings.LayerNorm.load_state_dict(tb.embeddings.LayerNorm.state_dict()) # no token-type embeddings15for j, i in enumerate([0, 2, 4, 7, 9, 11]): # the layers the released extraction script keeps16 t, s = tb.encoder.layer[i], sb.transformer.layer[j]17 for dst, src in [(s.attention.q_lin, t.attention.self.query), (s.attention.k_lin, t.attention.self.key),18 (s.attention.v_lin, t.attention.self.value), (s.attention.out_lin, t.attention.output.dense),19 (s.sa_layer_norm, t.attention.output.LayerNorm), (s.ffn.lin1, t.intermediate.dense),20 (s.ffn.lin2, t.output.dense), (s.output_layer_norm, t.output.LayerNorm)]:21 dst.load_state_dict(src.state_dict())22head = teacher.cls.predictions23student.vocab_transform.load_state_dict(head.transform.dense.state_dict())24student.vocab_layer_norm.load_state_dict(head.transform.LayerNorm.state_dict())25student.vocab_projector.bias.data.copy_(head.bias.data) # projector weight is tied to the word embeddings2627# ---- one training step on the triple loss ----28T, a_ce, a_mlm, a_cos = 2.0, 5.0, 2.0, 1.029collate = DataCollatorForLanguageModeling(tok, mlm_probability=0.15) # dynamic masking, no NSP30batch = collate([tok(t) for t in ["the cat sat on the mat .", "distillation makes models smaller ."]])31opt = torch.optim.AdamW(student.parameters(), lr=5e-4)3233with torch.no_grad():34 t_out = teacher(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"],35 output_hidden_states=True)36s_out = student(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"],37 labels=batch["labels"], output_hidden_states=True)3839keep = batch["attention_mask"].bool() # every non-padding position40s_logits, t_logits = s_out.logits[keep], t_out.logits[keep] # (n_tokens, 30522)41loss_ce = F.kl_div(F.log_softmax(s_logits / T, -1), F.softmax(t_logits / T, -1),42 reduction="batchmean") * T ** 2 # T²·KL(t || s)43loss_mlm = s_out.loss # cross-entropy on the masked positions only44h_s, h_t = s_out.hidden_states[-1][keep], t_out.hidden_states[-1][keep] # (n_tokens, 768)45loss_cos = F.cosine_embedding_loss(h_s, h_t, torch.ones(h_s.size(0))) # mean of 1 − cos4647loss = a_ce * loss_ce + a_mlm * loss_mlm + a_cos * loss_cos48opt.zero_grad(); loss.backward(); opt.step()49print(sum(p.numel() for p in student.parameters()) / 1e6) # about 66M (tied projector counted once)Use when
- +40% fewer parameters and 60% faster inference for 97% of BERT-base's GLUE score
- +Drop-in replacement: same tokenizer, width and fine-tuning recipe as BERT
- +Cheap to produce: about 90 hours on 8 V100s, using a teacher that already exists
- +General-purpose: one distillation run serves every downstream task
Avoid when
- −Gaps remain on harder tasks: 2.7 F1 behind BERT-base on SQuAD v1.1 without task-specific distillation
- −Needs the teacher in the loop: every training step pays a forward pass of the 12-layer model
- −The student is tied to the teacher's width and tokenizer by the layer-copy initialisation
- −Matches only final outputs and last hidden states; later methods distil intermediate layers and attention too
Where it's used
Low-latency classification
sentiment, intent and toxicity classifiers in production, where 6 layers instead of 12 roughly halves serving cost
On-device NLP
question answering and classification on phones; the paper's iPhone 7 Plus demo ran 71% faster than BERT-base
Embedding and retrieval baselines
a cheap encoder for sentence embeddings and re-ranking, often fine-tuned further
Check yourself
Why does the distillation loss multiply KL(t‖s) by T²?
Check yourself
In the DistilBERT ablation, which change hurt the GLUE macro-score the most?
Remember
- ✓DistilBERT: 6 layers, , 12 heads, 66M parameters, distilled from BERT-base during pre-training on Wikipedia and BookCorpus.
- ✓Triple loss: , with soft targets for teacher and student, BERT's MLM loss, and on the last hidden states.
- ✓Temperature exposes the teacher's ranking of wrong answers; the factor keeps the soft term's gradient from vanishing.
- ✓Initialisation from one teacher layer out of two is the most important ingredient (−3.69 GLUE points without it).
- ✓Result: 40% smaller, 60% faster, 97% of BERT-base's GLUE macro-score (77.0 vs 79.5).