Knowledge distillation

Knowledge distillation trains a small student to match a large teacher's outputs. Hinton et al. (2015) use α·T²·KL(softmax(v/T) ‖ softmax(z/T)) plus cross-entropy on the label. For LLMs, variants match tokens, whole teacher sequences (Kim & Rush 2016) or student samples under reverse KL.

15 min read1223 wordscore0 viewsupdated 2026-10-13

Definition

Knowledge distillation trains a small student model to reproduce the behaviour of a large teacher (or an ensemble) rather than only the training labels. In its standard form, from Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network" (NIPS 2014 Deep Learning Workshop, arXiv 2015), the student minimises the KL divergence between the teacher's and its own output distributions, both softened by a temperature T>1T>1, usually mixed with ordinary cross-entropy on the true labels. The idea goes back to Buciluǎ, Caruana and Niculescu-Mizil, "Model Compression" (KDD 2006).

Large models and ensembles are accurate but expensive to serve. Buciluǎ et al. (2006) labelled large amounts of synthetic data with an ensemble and trained a single small network on those labels, reaching nearly the ensemble's accuracy. Ba and Caruana ("Do Deep Nets Really Need to be Deep?", NIPS 2014) trained shallow nets to regress a deep net's logits with an L2 loss. Hinton et al. generalised both: match softened probabilities, and in the high-temperature limit you recover logit matching.

Their key point concerns what the teacher knows beyond its top answer. A teacher that says "2" with probability 0.99 may give "3" a probability of 10−610^{-6} and "7" one of 10−910^{-9}. Those tiny numbers say which wrong answers look like the right one, and so encode how the teacher generalises. Hinton called this dark knowledge. At T=1T=1 it barely affects the cross-entropy because the probabilities are so small. Raising the temperature brings it into view.

Soft targets and the loss

pi(T)=exp⁡(vi/T)∑jexp⁡(vj/T),qi(T)=exp⁡(zi/T)∑jexp⁡(zj/T),L=α T2 KL(p(T) ∥ q(T))+(1−α) CE(y, q(1))p_i^{(T)}=\frac{\exp(v_i/T)}{\sum_j\exp(v_j/T)},\qquad q_i^{(T)}=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)},\qquad \mathcal L=\alpha\,T^2\,\mathrm{KL}\big(p^{(T)}\,\|\,q^{(T)}\big)+(1-\alpha)\,\mathrm{CE}\big(y,\,q^{(1)}\big)
v, zv,\ z
teacher and student logits for the same input
TT
temperature; T=1T=1 is the ordinary softmax, larger TT flattens both distributions
p(T)p^{(T)}
soft targets: the teacher's softened distribution, fixed (no gradient)
KL(p∥q)\mathrm{KL}(p\|q)
=∑ipilog⁡(pi/qi)=\sum_ip_i\log(p_i/q_i); equal to the cross-entropy H(p,q)H(p,q) up to the constant teacher entropy, so the two give the same gradient
T2T^2
compensates for the 1/T21/T^2 shrinkage of the soft-target gradient, so α\alpha keeps its meaning when TT changes
α\alpha
mix of soft and hard targets; Hinton et al. found a considerably lower weight on the hard-label term worked best
∂ KL(p(T)∥q(T))∂zi=1T(qi−pi)  ≈  1NT2(zi−vi)for large T, zero-mean logits,T2 KL  →T→∞  12N∑i((zi−zˉ)−(vi−vˉ))2\frac{\partial\,\mathrm{KL}(p^{(T)}\|q^{(T)})}{\partial z_i}=\frac1T\big(q_i-p_i\big)\;\approx\;\frac1{NT^2}\big(z_i-v_i\big)\quad\text{for large }T\text{, zero-mean logits},\qquad T^2\,\mathrm{KL}\;\xrightarrow{T\to\infty}\;\frac1{2N}\sum_i\big((z_i-\bar z)-(v_i-\bar v)\big)^2
1T(qi−pi)\tfrac1T(q_i-p_i)
the gradient on a student logit: the softmax-cross-entropy gradient, divided by TT from the chain rule through z/Tz/T
1NT2(zi−vi)\tfrac1{NT^2}(z_i-v_i)
expanding ex/T≈1+x/Te^{x/T}\approx1+x/T over NN classes: the gradient becomes logit matching, as in Ba and Caruana, scaled by 1/T21/T^2
12N∑(⋅)2\tfrac1{2N}\sum(\cdot)^2
so at high TT, T2⋅T^2\cdotKL is a squared error on centred logits; at moderate TT very negative logits are matched less strictly, which Hinton et al. note can help
logit distillation (Hinton, Vinyals, Dean 2015): one training examplexinputteacher (frozen)large or an ensemblestudent (trained)smaller, fasterp = softmax(v/T)soft targetsq = softmax(z/T)same Tsoftmax(z)T = 1T²·KL(p ‖ q)soft loss, weight αCE(y, softmax(z))hard label, 1 − αloss∇ flows to student∂(T²·KL)/∂zᵢ = T·(qᵢ − pᵢ): the T² keeps soft and hard gradients on the same scale as T changeswhat the student imitateslogit KDHinton et al. 2015match softened outputsT²·KL(p_T ‖ q_T)feature KDFitNets, Romero et al. 2015regress a student layeronto a teacher hint layersequence-level KDKim & Rush 2016NLL on the teacher'sbeam-search output ŷon-policy KDMiniLLM, GKD (2024)student samples, teacherscores every tokenLLMs: SFT on teacher-written text is sequence-level KD (Alpaca: 52K examples; DeepSeek-R1-Distill: ~800K)forward KL(p‖q) on teacher text covers all teacher modes; reverse KL(q‖p) on student samples seeks one modeDistilBERT: 6 of BERT-base's 12 layers, 40% smaller, 60% faster, 97% of its GLUE score
fig.Logit distillation for one example. The input goes through the frozen teacher and the trainable student. Both sets of logits are divided by the same temperature T and softmaxed. The student is trained on T²·KL to the teacher's soft targets (weight α) plus cross-entropy on the hard label at T = 1 (weight 1 − α). The gradient flows only into the student. Below are the four families of what a student can imitate: output distributions, intermediate features, whole teacher-generated sequences, and the teacher's per-token judgement of the student's own samples.

Results and configurations

MNIST (Hinton et al.)
teacher 2×1200 ReLU units with dropout: 67 test errors. A 2×800 net trained on hard labels: 146 errors. The same net with soft targets at T=20T=20: 74 errors. With 30 units per layer, TT between 2.5 and 4 worked best
Missing class
with every 3 removed from the transfer set, the distilled MNIST net still made 206 errors, 133 on the 1010 test 3s; raising the bias of class 3 cut this to 109 errors, 14 on 3s
Speech (Hinton et al.)
one acoustic model: 58.9% frame accuracy, 10.9% WER; 10-model ensemble: 61.1%, 10.7%; distilled single model: 60.8%, 10.7%
Soft targets as regulariser
trained on 3% of the speech data, hard labels overfit to 44.5% test frame accuracy, while soft targets reached 57.0% (58.9% with all the data)
DistilBERT (Sanh et al. 2019)
6 layers instead of BERT-base's 12, initialised from alternate teacher layers; loss = soft-target CE + MLM + cosine loss on hidden states; 40% smaller, 60% faster, 97% of BERT's GLUE performance (see DistilBERT)
Gemma 2 (2024)
the 2B and 9B models were pre-trained with token-level distillation from a larger teacher instead of next-token prediction, on 2T and 8T tokens
Llama 3.2 1B / 3B (2024)
pruned from Llama 3.1 8B, then pre-trained with logits from Llama 3.1 8B and 70B as token-level targets
DeepSeek-R1-Distill (2025)
Qwen2.5 and Llama 3 models (1.5B–70B) fine-tuned with plain SFT on about 800K samples curated with DeepSeek-R1; the 32B student scores 72.6% pass@1 on AIME 2024, against 47.0% for the same base model trained with large-scale RL

Logit, feature and sequence-level distillation

Logit (response) distillation is the Hinton loss above, applied per output. For a language model the "output" is the next-token distribution at every position, so the loss is a sum over positions of a KL over the vocabulary. This needs the teacher's full logits, and both models must share a tokenizer, or the two distributions are over different vocabularies.

Feature distillation matches internal representations. FitNets (Romero, Ballas, Kahou, Chassang, Gatta and Bengio, ICLR 2015) trains a thin, deep student so that one of its middle layers, passed through a learned regressor, predicts a "hint" layer of the teacher. Later work matches attention maps (Zagoruyko and Komodakis, 2017). In transformers, DistilBERT adds a cosine loss on hidden states, and TinyBERT matches attention matrices and hidden states layer by layer.

Sequence-level distillation (Kim and Rush, "Sequence-Level Knowledge Distillation", EMNLP 2016) matches the teacher's distribution over whole sequences. That distribution is intractable, so they approximate it by its mode: run beam search with the teacher and train the student with plain NLL on the output y^\hat y. On neural machine translation it beat word-level KD. It let the student decode greedily with little loss, so beam search was no longer needed. Their best student ran about 10 times faster than the teacher with little loss in BLEU. They also proposed word-level KD per token and sequence-level interpolation, which trains on the beam candidate closest to the reference.

Distilling LLMs: data, divergence, and whose samples

The most common form of LLM distillation is sequence-level: generate text with a strong teacher and run Supervised fine-tuning (SFT) of the student on it. Stanford's Alpaca (2023) fine-tuned LLaMA 7B on 52K instruction–response pairs generated by text-davinci-003 with self-instruct. Orca (Mukherjee et al., 2023) trained on GPT-4 explanation traces rather than bare answers. DeepSeek-R1's distilled models used about 800K samples curated with R1, including long reasoning traces, and no RL. This works through an API that returns only text. It also copies the teacher's mistakes and style, and model licences sometimes forbid it.

With access to the teacher's logits, the choice of divergence matters. Forward KL KL(p∥q)\mathrm{KL}(p\|q) weights errors by the teacher's probabilities, so the student must put mass everywhere the teacher does. It is mode-covering. A low-capacity student that cannot match a multi-modal teacher spreads its mass, which leaves probability on text the teacher would never write. Reverse KL KL(q∥p)\mathrm{KL}(q\|p) weights errors by the student's own probabilities. It is mode-seeking: the student may ignore some teacher modes but is punished for producing what the teacher finds unlikely.

MiniLLM (Gu, Dong, Wei and Huang, ICLR 2024) minimises reverse KL at the sequence level with a policy-gradient method and reports more precise, better-calibrated generations than forward-KL KD. GKD, generalized knowledge distillation (Agarwal et al., ICLR 2024), tackles the train–inference mismatch. A student trained only on teacher text never sees its own mistakes, the same exposure bias as Teacher forcing. GKD samples outputs from the student and has the teacher score every token, with a generalised Jensen–Shannon divergence that interpolates between forward and reverse KL.

“The cat sat on the ___” · loss = α·T²·KL(p‖q) + (1 − α)·CE(hard label “mat”)T = 4.0: ■ teacher p = softmax(v/T) ■ student q = softmax(z/T)mat*.266.256+0.194floor.178.094+0.162sofa.157.200−0.204bed.146.085+0.117roof.103.148−0.126car.067.102−0.077moon.046.060−0.029idea.036.054−0.038logit update −∂loss/∂zᵢ (raise / lower), * = hard labelvs T (log axes): KL(p‖q) · T²·KL(p‖q)‖gradient on z‖: raw · ×T² · hard CE125102011e−21e−411e−21e−4KL(p‖q) at T = 4.0: 0.0719 · ×T² = 1.1505 · large-T limit (1/2N)·Σ(centred logit gap)² = 1.2655‖∂KL(p‖q)/∂z‖ = 0.0320 raw, 0.5117 with T² · hard-label CE = 0.431, ‖∂CE/∂z‖ = 0.431total = 0.50·1.151 + 0.50·0.431 = 0.791 · dark knowledge: teacher mass off “mat” = 0.734on “floor”: T²·KL term +0.336 (teacher’s 2nd choice) · hard CE −0.012 (one-hot: all wrong tokens alike)forward KL(p‖q) weights errors by the teacher's p: the student must cover everything the teacher allowswithout T², the soft gradient shrinks like 1/T²: 0.220 at T = 1, 0.0014 at T = 20teacher logits v = (5, 3.4, 2.9, 2.6, 1.2, −0.5, −2, −3); student z blended 0% toward v
divergence
fig.One next-token prediction, "The cat sat on the ___", over 8 candidate tokens with fixed teacher logits v = (5, 3.4, 2.9, 2.6, 1.2, −0.5, −2, −3). The student agrees on "mat" but ranks "sofa" above "floor". At T = 1 the teacher puts 0.140 on "floor" against the student's 0.012, and KL(p‖q) = 0.4018. At the default T = 4 the teacher is much flatter (0.266 on "mat"; 0.734 of its mass, the dark knowledge, lies elsewhere). KL drops to 0.0719, but T²·KL = 1.1505, close to its large-T limit 1.2655, the centred logit-matching loss. The gradient norm is 0.0320 without the T² factor and 0.5117 with it. The lower-right plot shows the raw gradient collapsing from 0.220 at T = 1 to 0.0014 at T = 20, while the scaled one stays near the hard-label gradient (0.431). With α = 0.5 the loss is 0.5·1.151 + 0.5·0.431 = 0.791. The soft term raises "floor" (+0.336) because the teacher ranks it second. The hard label lowers it (−0.012), because one-hot targets treat every wrong token alike. Reverse KL at T = 1 is 0.3027. It puts its largest push on the student's own top token and barely raises "floor" (+0.033, against +0.128 under forward KL), which the student already treats as unlikely. Blending the student 50% toward the teacher cuts T²·KL at T = 4 to 0.2978.

Worked example: four tokens at T = 3

Teacher logits v=(4,2,1,−1)v=(4,2,1,-1) and student logits z=(3,0,1.5,−0.5)z=(3,0,1.5,-0.5) for the tokens (mat, floor, sofa, idea). The hard label is "mat". Use T=3T=3 and α=0.7\alpha=0.7.

tokenp at T=1p at T=3q at T=3∂(T²KL)/∂z = T(q − p)∂CE/∂z (T=1)
mat (label)0.83900.48310.4375−0.1367−0.2326
floor0.11350.24800.1609−0.2612+0.0382
sofa0.04180.17770.2653+0.2629+0.1712
idea0.00570.09120.1362+0.1350+0.0232
Soft and hard terms (computed in Python with numpy)

Assembling the loss

  1. 1Softening: at T=1T=1 the teacher gives 0.161 to tokens other than "mat"; at T=3T=3 it gives 0.517. The ratio floor/sofa falls from e1=2.718e^{1}=2.718 to e1/3=1.396e^{1/3}=1.396. The ranking is kept, but the small probabilities become large enough to train on.
  2. 2Soft term: KL(p(3)∥q(3))=0.0473\mathrm{KL}(p^{(3)}\|q^{(3)})=0.0473, times T2=9T^2=9: 0.42560.4256. Its gradient T(q−p)T(q-p) is largest on "floor" (−0.2612-0.2612, so raise it) and "sofa" (+0.2629+0.2629, so lower it): exactly the ranking error the student makes.
  3. 3Hard term: CE=−ln⁡qmat(1)=0.2648\mathrm{CE}=-\ln q^{(1)}_{mat}=0.2648. Its gradient is q(1)−onehotq^{(1)}-\text{onehot}, which lowers "floor" (+0.0382+0.0382) along with every other wrong token. The label cannot say that "floor" is a reasonable answer.
  4. 4Total: 0.7⋅0.4256+0.3⋅0.2648=0.37730.7\cdot0.4256+0.3\cdot0.2648=0.3773, with gradient (−0.1655,−0.1714,+0.2354,+0.1014)(-0.1655,-0.1714,+0.2354,+0.1014). "floor" ends up being raised.
  5. 5High-temperature check: T2KLT^2\mathrm{KL} is 0.53360.5336 at T=10T=10 and 0.56020.5602 at T=100T=100, approaching 12N∑((z−zˉ)−(v−vˉ))2=0.5625\tfrac1{2N}\sum((z-\bar z)-(v-\bar v))^2=0.5625. The scaled gradient approaches ((z−zˉ)−(v−vˉ))/N=(−0.125,−0.375,0.25,0.25)((z-\bar z)-(v-\bar v))/N=(-0.125,-0.375,0.25,0.25), which is logit matching.
python
1import torch, torch.nn.functional as F
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4def kd_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5, ignore_index=-100):
5 """student/teacher_logits: (B, L, V) predicting labels (B, L); labels = -100 on prompt and pad."""
6 mask = labels != ignore_index
7 V = min(student_logits.size(-1), teacher_logits.size(-1)) # guard against padded vocabularies
8 s, t = student_logits[mask][:, :V].float(), teacher_logits[mask][:, :V].float()
9 soft = F.kl_div(F.log_softmax(s / T, -1), F.log_softmax(t / T, -1),
10 log_target=True, reduction="batchmean") * T * T # T^2 · KL(p_T ‖ q_T) per token
11 hard = F.cross_entropy(s, labels[mask])
12 return alpha * soft + (1 - alpha) * hard
13
14# check on the worked example: one position, vocabulary (mat, floor, sofa, idea), label "mat"
15z = torch.tensor([[[3.0, 0.0, 1.5, -0.5]]]); v = torch.tensor([[[4.0, 2.0, 1.0, -1.0]]])
16print(kd_loss(z, v, torch.tensor([[0]]), T=3.0, alpha=0.7)) # 0.3773
17
18# distilling a Llama 3.1 8B teacher into Llama 3.2 1B (same 128,256-token vocabulary)
19tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
20teacher = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct",
21 torch_dtype=torch.bfloat16).eval().requires_grad_(False)
22student = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", torch_dtype=torch.bfloat16)
23opt = torch.optim.AdamW(student.parameters(), lr=1e-5)
24
25def step(input_ids, attention_mask, labels): # labels already shifted: labels[:, t] = input_ids[:, t+1]
26 with torch.no_grad():
27 t_logits = teacher(input_ids=input_ids, attention_mask=attention_mask).logits
28 s_logits = student(input_ids=input_ids, attention_mask=attention_mask).logits
29 loss = kd_loss(s_logits, t_logits, labels, T=2.0, alpha=0.5)
30 opt.zero_grad(); loss.backward(); opt.step()
31 return loss.item()
32
33# sequence-level KD (Kim & Rush): let the teacher write the targets, then ordinary SFT on them
34prompt = tok.apply_chat_template([{"role": "user", "content": "Explain KV caching in two sentences."}],
35 add_generation_prompt=True, return_tensors="pt")
36y_hat = teacher.generate(prompt, max_new_tokens=128, num_beams=4, do_sample=False)
37# ...add (prompt, y_hat) to the SFT set and train the student with plain cross-entropy
Token-level KD for causal LMs in PyTorch (Hinton loss over every supervised position), and a check on the worked example (written for PyTorch 2; the numbers in comments were computed with numpy)

Use when

  • +Small, fast students that keep most of the teacher's quality (DistilBERT: 40% smaller, 97% of GLUE)
  • +Soft targets regularise and carry more information per example than labels
  • +Sequence-level KD needs only the teacher's text, so it works through an API
  • +Combines with Quantization and Pruning (Llama 3.2 1B/3B were pruned, then distilled)

Avoid when

  • −Logit KD needs the teacher's full output distribution at every position, which is expensive at LLM vocabulary sizes
  • −Requires a shared tokenizer for token-level KL
  • −The student inherits the teacher's errors and biases, and cannot exceed it on the distilled task
  • −Off-policy training leaves exposure bias; on-policy KD costs student sampling plus teacher scoring

Where it's used

Compressing encoders

DistilBERT and TinyBERT for classification and retrieval at a fraction of BERT's latency

Small LLM pre-training

Gemma 2 2B/9B and Llama 3.2 1B/3B trained on teacher logits

Reasoning transfer

DeepSeek-R1-Distill: SFT on long reasoning traces from a large RL-trained model

Faster decoding

Kim and Rush's students decode greedily, with no beam search; distilled draft models also serve Speculative decoding

Check yourself

Why does the Hinton loss multiply the soft-target KL by T²?

Check yourself

A 1B student is distilled from a 70B teacher whose next-token distributions are often multi-modal. Which objective tends to avoid giving probability to text the teacher would not produce?

Remember

  • ✓Hinton loss: αT2 KL(softmax(v/T)∥softmax(z/T))+(1−α) CE(y,softmax(z))\alpha T^2\,\mathrm{KL}(\mathrm{softmax}(v/T)\|\mathrm{softmax}(z/T))+(1-\alpha)\,\mathrm{CE}(y,\mathrm{softmax}(z)); the gradient on ziz_i is T(qi−pi)T(q_i-p_i) after the T2T^2.
  • ✓Temperature exposes dark knowledge, the teacher's ranking of wrong answers. As T→∞T\to\infty, T2⋅T^2\cdotKL becomes squared error on centred logits (Ba and Caruana's logit matching).
  • ✓Three things to imitate: output distributions (logit KD), hidden features (FitNets, DistilBERT, TinyBERT), or whole teacher sequences (Kim & Rush sequence-level KD, i.e. SFT on teacher outputs).
  • ✓For LLMs: forward KL is mode-covering, reverse KL (MiniLLM) is mode-seeking, and on-policy KD (GKD) trains on student samples to remove exposure bias.
  • ✓Practice: Gemma 2 and Llama 3.2 small models were pre-trained on teacher logits; DeepSeek-R1-Distill-Qwen-32B reached 72.6% on AIME 2024 from SFT on R1 samples alone.
← back to Deep Learning roadmap