Definition
Knowledge distillation trains a small student model to reproduce the behaviour of a large teacher (or an ensemble) rather than only the training labels. In its standard form, from Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network" (NIPS 2014 Deep Learning Workshop, arXiv 2015), the student minimises the KL divergence between the teacher's and its own output distributions, both softened by a temperature , usually mixed with ordinary cross-entropy on the true labels. The idea goes back to Buciluǎ, Caruana and Niculescu-Mizil, "Model Compression" (KDD 2006).
Large models and ensembles are accurate but expensive to serve. Buciluǎ et al. (2006) labelled large amounts of synthetic data with an ensemble and trained a single small network on those labels, reaching nearly the ensemble's accuracy. Ba and Caruana ("Do Deep Nets Really Need to be Deep?", NIPS 2014) trained shallow nets to regress a deep net's logits with an L2 loss. Hinton et al. generalised both: match softened probabilities, and in the high-temperature limit you recover logit matching.
Their key point concerns what the teacher knows beyond its top answer. A teacher that says "2" with probability 0.99 may give "3" a probability of and "7" one of . Those tiny numbers say which wrong answers look like the right one, and so encode how the teacher generalises. Hinton called this dark knowledge. At it barely affects the cross-entropy because the probabilities are so small. Raising the temperature brings it into view.
Soft targets and the loss
- teacher and student logits for the same input
- temperature; is the ordinary softmax, larger flattens both distributions
- soft targets: the teacher's softened distribution, fixed (no gradient)
- ; equal to the cross-entropy up to the constant teacher entropy, so the two give the same gradient
- compensates for the shrinkage of the soft-target gradient, so keeps its meaning when changes
- mix of soft and hard targets; Hinton et al. found a considerably lower weight on the hard-label term worked best
- the gradient on a student logit: the softmax-cross-entropy gradient, divided by from the chain rule through
- expanding over classes: the gradient becomes logit matching, as in Ba and Caruana, scaled by
- so at high , KL is a squared error on centred logits; at moderate very negative logits are matched less strictly, which Hinton et al. note can help
Results and configurations
- MNIST (Hinton et al.)
- teacher 2×1200 ReLU units with dropout: 67 test errors. A 2×800 net trained on hard labels: 146 errors. The same net with soft targets at : 74 errors. With 30 units per layer, between 2.5 and 4 worked best
- Missing class
- with every 3 removed from the transfer set, the distilled MNIST net still made 206 errors, 133 on the 1010 test 3s; raising the bias of class 3 cut this to 109 errors, 14 on 3s
- Speech (Hinton et al.)
- one acoustic model: 58.9% frame accuracy, 10.9% WER; 10-model ensemble: 61.1%, 10.7%; distilled single model: 60.8%, 10.7%
- Soft targets as regulariser
- trained on 3% of the speech data, hard labels overfit to 44.5% test frame accuracy, while soft targets reached 57.0% (58.9% with all the data)
- DistilBERT (Sanh et al. 2019)
- 6 layers instead of BERT-base's 12, initialised from alternate teacher layers; loss = soft-target CE + MLM + cosine loss on hidden states; 40% smaller, 60% faster, 97% of BERT's GLUE performance (see DistilBERT)
- Gemma 2 (2024)
- the 2B and 9B models were pre-trained with token-level distillation from a larger teacher instead of next-token prediction, on 2T and 8T tokens
- Llama 3.2 1B / 3B (2024)
- pruned from Llama 3.1 8B, then pre-trained with logits from Llama 3.1 8B and 70B as token-level targets
- DeepSeek-R1-Distill (2025)
- Qwen2.5 and Llama 3 models (1.5B–70B) fine-tuned with plain SFT on about 800K samples curated with DeepSeek-R1; the 32B student scores 72.6% pass@1 on AIME 2024, against 47.0% for the same base model trained with large-scale RL
Logit, feature and sequence-level distillation
Logit (response) distillation is the Hinton loss above, applied per output. For a language model the "output" is the next-token distribution at every position, so the loss is a sum over positions of a KL over the vocabulary. This needs the teacher's full logits, and both models must share a tokenizer, or the two distributions are over different vocabularies.
Feature distillation matches internal representations. FitNets (Romero, Ballas, Kahou, Chassang, Gatta and Bengio, ICLR 2015) trains a thin, deep student so that one of its middle layers, passed through a learned regressor, predicts a "hint" layer of the teacher. Later work matches attention maps (Zagoruyko and Komodakis, 2017). In transformers, DistilBERT adds a cosine loss on hidden states, and TinyBERT matches attention matrices and hidden states layer by layer.
Sequence-level distillation (Kim and Rush, "Sequence-Level Knowledge Distillation", EMNLP 2016) matches the teacher's distribution over whole sequences. That distribution is intractable, so they approximate it by its mode: run beam search with the teacher and train the student with plain NLL on the output . On neural machine translation it beat word-level KD. It let the student decode greedily with little loss, so beam search was no longer needed. Their best student ran about 10 times faster than the teacher with little loss in BLEU. They also proposed word-level KD per token and sequence-level interpolation, which trains on the beam candidate closest to the reference.
Distilling LLMs: data, divergence, and whose samples
The most common form of LLM distillation is sequence-level: generate text with a strong teacher and run Supervised fine-tuning (SFT) of the student on it. Stanford's Alpaca (2023) fine-tuned LLaMA 7B on 52K instruction–response pairs generated by text-davinci-003 with self-instruct. Orca (Mukherjee et al., 2023) trained on GPT-4 explanation traces rather than bare answers. DeepSeek-R1's distilled models used about 800K samples curated with R1, including long reasoning traces, and no RL. This works through an API that returns only text. It also copies the teacher's mistakes and style, and model licences sometimes forbid it.
With access to the teacher's logits, the choice of divergence matters. Forward KL weights errors by the teacher's probabilities, so the student must put mass everywhere the teacher does. It is mode-covering. A low-capacity student that cannot match a multi-modal teacher spreads its mass, which leaves probability on text the teacher would never write. Reverse KL weights errors by the student's own probabilities. It is mode-seeking: the student may ignore some teacher modes but is punished for producing what the teacher finds unlikely.
MiniLLM (Gu, Dong, Wei and Huang, ICLR 2024) minimises reverse KL at the sequence level with a policy-gradient method and reports more precise, better-calibrated generations than forward-KL KD. GKD, generalized knowledge distillation (Agarwal et al., ICLR 2024), tackles the train–inference mismatch. A student trained only on teacher text never sees its own mistakes, the same exposure bias as Teacher forcing. GKD samples outputs from the student and has the teacher score every token, with a generalised Jensen–Shannon divergence that interpolates between forward and reverse KL.
Worked example: four tokens at T = 3
Teacher logits and student logits for the tokens (mat, floor, sofa, idea). The hard label is "mat". Use and .
| token | p at T=1 | p at T=3 | q at T=3 | ∂(T²KL)/∂z = T(q − p) | ∂CE/∂z (T=1) |
|---|---|---|---|---|---|
| mat (label) | 0.8390 | 0.4831 | 0.4375 | −0.1367 | −0.2326 |
| floor | 0.1135 | 0.2480 | 0.1609 | −0.2612 | +0.0382 |
| sofa | 0.0418 | 0.1777 | 0.2653 | +0.2629 | +0.1712 |
| idea | 0.0057 | 0.0912 | 0.1362 | +0.1350 | +0.0232 |
Assembling the loss
- 1Softening: at the teacher gives 0.161 to tokens other than "mat"; at it gives 0.517. The ratio floor/sofa falls from to . The ranking is kept, but the small probabilities become large enough to train on.
- 2Soft term: , times : . Its gradient is largest on "floor" (, so raise it) and "sofa" (, so lower it): exactly the ranking error the student makes.
- 3Hard term: . Its gradient is , which lowers "floor" () along with every other wrong token. The label cannot say that "floor" is a reasonable answer.
- 4Total: , with gradient . "floor" ends up being raised.
- 5High-temperature check: is at and at , approaching . The scaled gradient approaches , which is logit matching.
1import torch, torch.nn.functional as F2from transformers import AutoModelForCausalLM, AutoTokenizer34def kd_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5, ignore_index=-100):5 """student/teacher_logits: (B, L, V) predicting labels (B, L); labels = -100 on prompt and pad."""6 mask = labels != ignore_index7 V = min(student_logits.size(-1), teacher_logits.size(-1)) # guard against padded vocabularies8 s, t = student_logits[mask][:, :V].float(), teacher_logits[mask][:, :V].float()9 soft = F.kl_div(F.log_softmax(s / T, -1), F.log_softmax(t / T, -1),10 log_target=True, reduction="batchmean") * T * T # T^2 · KL(p_T ‖ q_T) per token11 hard = F.cross_entropy(s, labels[mask])12 return alpha * soft + (1 - alpha) * hard1314# check on the worked example: one position, vocabulary (mat, floor, sofa, idea), label "mat"15z = torch.tensor([[[3.0, 0.0, 1.5, -0.5]]]); v = torch.tensor([[[4.0, 2.0, 1.0, -1.0]]])16print(kd_loss(z, v, torch.tensor([[0]]), T=3.0, alpha=0.7)) # 0.37731718# distilling a Llama 3.1 8B teacher into Llama 3.2 1B (same 128,256-token vocabulary)19tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")20teacher = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct",21 torch_dtype=torch.bfloat16).eval().requires_grad_(False)22student = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", torch_dtype=torch.bfloat16)23opt = torch.optim.AdamW(student.parameters(), lr=1e-5)2425def step(input_ids, attention_mask, labels): # labels already shifted: labels[:, t] = input_ids[:, t+1]26 with torch.no_grad():27 t_logits = teacher(input_ids=input_ids, attention_mask=attention_mask).logits28 s_logits = student(input_ids=input_ids, attention_mask=attention_mask).logits29 loss = kd_loss(s_logits, t_logits, labels, T=2.0, alpha=0.5)30 opt.zero_grad(); loss.backward(); opt.step()31 return loss.item()3233# sequence-level KD (Kim & Rush): let the teacher write the targets, then ordinary SFT on them34prompt = tok.apply_chat_template([{"role": "user", "content": "Explain KV caching in two sentences."}],35 add_generation_prompt=True, return_tensors="pt")36y_hat = teacher.generate(prompt, max_new_tokens=128, num_beams=4, do_sample=False)37# ...add (prompt, y_hat) to the SFT set and train the student with plain cross-entropyUse when
- +Small, fast students that keep most of the teacher's quality (DistilBERT: 40% smaller, 97% of GLUE)
- +Soft targets regularise and carry more information per example than labels
- +Sequence-level KD needs only the teacher's text, so it works through an API
- +Combines with Quantization and Pruning (Llama 3.2 1B/3B were pruned, then distilled)
Avoid when
- −Logit KD needs the teacher's full output distribution at every position, which is expensive at LLM vocabulary sizes
- −Requires a shared tokenizer for token-level KL
- −The student inherits the teacher's errors and biases, and cannot exceed it on the distilled task
- −Off-policy training leaves exposure bias; on-policy KD costs student sampling plus teacher scoring
Where it's used
Compressing encoders
DistilBERT and TinyBERT for classification and retrieval at a fraction of BERT's latency
Small LLM pre-training
Gemma 2 2B/9B and Llama 3.2 1B/3B trained on teacher logits
Reasoning transfer
DeepSeek-R1-Distill: SFT on long reasoning traces from a large RL-trained model
Faster decoding
Kim and Rush's students decode greedily, with no beam search; distilled draft models also serve Speculative decoding
Check yourself
Why does the Hinton loss multiply the soft-target KL by T²?
Check yourself
A 1B student is distilled from a 70B teacher whose next-token distributions are often multi-modal. Which objective tends to avoid giving probability to text the teacher would not produce?
Remember
- ✓Hinton loss: ; the gradient on is after the .
- ✓Temperature exposes dark knowledge, the teacher's ranking of wrong answers. As , KL becomes squared error on centred logits (Ba and Caruana's logit matching).
- ✓Three things to imitate: output distributions (logit KD), hidden features (FitNets, DistilBERT, TinyBERT), or whole teacher sequences (Kim & Rush sequence-level KD, i.e. SFT on teacher outputs).
- ✓For LLMs: forward KL is mode-covering, reverse KL (MiniLLM) is mode-seeking, and on-policy KD (GKD) trains on student samples to remove exposure bias.
- ✓Practice: Gemma 2 and Llama 3.2 small models were pre-trained on teacher logits; DeepSeek-R1-Distill-Qwen-32B reached 72.6% on AIME 2024 from SFT on R1 samples alone.