# Knowledge distillation

> Knowledge distillation trains a small student to match a large teacher's outputs. Hinton et al. (2015) use α·T²·KL(softmax(v/T) ‖ softmax(z/T)) plus cross-entropy on the label. For LLMs, variants match tokens, whole teacher sequences (Kim & Rush 2016) or student samples under reverse KL.

Source: https://www.mlfoundry.in/learn/knowledge-distillation  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Knowledge distillation:** trains a small **student** model to reproduce the behaviour of a large **teacher** (or an ensemble) rather than only the training labels. In its standard form, from Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network" (NIPS 2014 Deep Learning Workshop, arXiv 2015), the student minimises the KL divergence between the teacher's and its own output distributions, both softened by a **temperature** $T>1$, usually mixed with ordinary cross-entropy on the true labels. The idea goes back to Buciluǎ, Caruana and Niculescu-Mizil, "Model Compression" (KDD 2006).

Large models and ensembles are accurate but expensive to serve. Buciluǎ et al. (2006) labelled large amounts of synthetic data with an ensemble and trained a single small network on those labels, reaching nearly the ensemble's accuracy. Ba and Caruana ("Do Deep Nets Really Need to be Deep?", NIPS 2014) trained shallow nets to regress a deep net's **logits** with an L2 loss. Hinton et al. generalised both: match softened probabilities, and in the high-temperature limit you recover logit matching.

Their key point concerns what the teacher knows beyond its top answer. A teacher that says "2" with probability 0.99 may give "3" a probability of $10^{-6}$ and "7" one of $10^{-9}$. Those tiny numbers say which wrong answers look like the right one, and so encode how the teacher generalises. Hinton called this **dark knowledge**. At $T=1$ it barely affects the cross-entropy because the probabilities are so small. Raising the temperature brings it into view.

## Soft targets and the loss

$$
p_i^{(T)}=\frac{\exp(v_i/T)}{\sum_j\exp(v_j/T)},\qquad q_i^{(T)}=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)},\qquad \mathcal L=\alpha\,T^2\,\mathrm{KL}\big(p^{(T)}\,\|\,q^{(T)}\big)+(1-\alpha)\,\mathrm{CE}\big(y,\,q^{(1)}\big)
$$

- $v,\ z$: teacher and student logits for the same input
- $T$: temperature; $T=1$ is the ordinary softmax, larger $T$ flattens both distributions
- $p^{(T)}$: **soft targets**: the teacher's softened distribution, fixed (no gradient)
- $\mathrm{KL}(p\|q)$: $=\sum_ip_i\log(p_i/q_i)$; equal to the cross-entropy $H(p,q)$ up to the constant teacher entropy, so the two give the same gradient
- $T^2$: compensates for the $1/T^2$ shrinkage of the soft-target gradient, so $\alpha$ keeps its meaning when $T$ changes
- $\alpha$: mix of soft and hard targets; Hinton et al. found a considerably lower weight on the hard-label term worked best

$$
\frac{\partial\,\mathrm{KL}(p^{(T)}\|q^{(T)})}{\partial z_i}=\frac1T\big(q_i-p_i\big)\;\approx\;\frac1{NT^2}\big(z_i-v_i\big)\quad\text{for large }T\text{, zero-mean logits},\qquad T^2\,\mathrm{KL}\;\xrightarrow{T\to\infty}\;\frac1{2N}\sum_i\big((z_i-\bar z)-(v_i-\bar v)\big)^2
$$

- $\tfrac1T(q_i-p_i)$: the gradient on a student logit: the softmax-cross-entropy gradient, divided by $T$ from the chain rule through $z/T$
- $\tfrac1{NT^2}(z_i-v_i)$: expanding $e^{x/T}\approx1+x/T$ over $N$ classes: the gradient becomes logit matching, as in Ba and Caruana, scaled by $1/T^2$
- $\tfrac1{2N}\sum(\cdot)^2$: so at high $T$, $T^2\cdot$KL is a squared error on centred logits; at moderate $T$ very negative logits are matched less strictly, which Hinton et al. note can help

> Figure: Logit distillation for one example. The input goes through the frozen teacher and the trainable student. Both sets of logits are divided by the same temperature T and softmaxed. The student is trained on T²·KL to the teacher's soft targets (weight α) plus cross-entropy on the hard label at T = 1 (weight 1 − α). The gradient flows only into the student. Below are the four families of what a student can imitate: output distributions, intermediate features, whole teacher-generated sequences, and the teacher's per-token judgement of the student's own samples.

**Results and configurations**

- **MNIST (Hinton et al.):** teacher 2×1200 ReLU units with dropout: 67 test errors. A 2×800 net trained on hard labels: 146 errors. The same net with soft targets at $T=20$: 74 errors. With 30 units per layer, $T$ between 2.5 and 4 worked best
- **Missing class:** with every 3 removed from the transfer set, the distilled MNIST net still made 206 errors, 133 on the 1010 test 3s; raising the bias of class 3 cut this to 109 errors, 14 on 3s
- **Speech (Hinton et al.):** one acoustic model: 58.9% frame accuracy, 10.9% WER; 10-model ensemble: 61.1%, 10.7%; distilled single model: 60.8%, 10.7%
- **Soft targets as regulariser:** trained on 3% of the speech data, hard labels overfit to 44.5% test frame accuracy, while soft targets reached 57.0% (58.9% with all the data)
- **DistilBERT (Sanh et al. 2019):** 6 layers instead of BERT-base's 12, initialised from alternate teacher layers; loss = soft-target CE + MLM + cosine loss on hidden states; 40% smaller, 60% faster, 97% of BERT's GLUE performance (see **DistilBERT**)
- **Gemma 2 (2024):** the 2B and 9B models were pre-trained with token-level distillation from a larger teacher instead of next-token prediction, on 2T and 8T tokens
- **Llama 3.2 1B / 3B (2024):** pruned from Llama 3.1 8B, then pre-trained with logits from Llama 3.1 8B and 70B as token-level targets
- **DeepSeek-R1-Distill (2025):** Qwen2.5 and Llama 3 models (1.5B–70B) fine-tuned with plain SFT on about 800K samples curated with DeepSeek-R1; the 32B student scores 72.6% pass@1 on AIME 2024, against 47.0% for the same base model trained with large-scale RL

## Logit, feature and sequence-level distillation

**Logit (response) distillation** is the Hinton loss above, applied per output. For a language model the "output" is the next-token distribution at every position, so the loss is a sum over positions of a KL over the vocabulary. This needs the teacher's full logits, and both models must share a tokenizer, or the two distributions are over different vocabularies.

**Feature distillation** matches internal representations. FitNets (Romero, Ballas, Kahou, Chassang, Gatta and Bengio, ICLR 2015) trains a thin, deep student so that one of its middle layers, passed through a learned regressor, predicts a "hint" layer of the teacher. Later work matches attention maps (Zagoruyko and Komodakis, 2017). In transformers, DistilBERT adds a cosine loss on hidden states, and TinyBERT matches attention matrices and hidden states layer by layer.

**Sequence-level distillation** (Kim and Rush, "Sequence-Level Knowledge Distillation", EMNLP 2016) matches the teacher's distribution over *whole sequences*. That distribution is intractable, so they approximate it by its mode: run beam search with the teacher and train the student with plain NLL on the output $\hat y$. On neural machine translation it beat word-level KD. It let the student decode greedily with little loss, so beam search was no longer needed. Their best student ran about 10 times faster than the teacher with little loss in BLEU. They also proposed word-level KD per token and **sequence-level interpolation**, which trains on the beam candidate closest to the reference.

## Distilling LLMs: data, divergence, and whose samples

The most common form of LLM distillation is sequence-level: generate text with a strong teacher and run **Supervised fine-tuning (SFT)** of the student on it. Stanford's Alpaca (2023) fine-tuned LLaMA 7B on 52K instruction–response pairs generated by `text-davinci-003` with self-instruct. Orca (Mukherjee et al., 2023) trained on GPT-4 explanation traces rather than bare answers. DeepSeek-R1's distilled models used about 800K samples curated with R1, including long reasoning traces, and no RL. This works through an API that returns only text. It also copies the teacher's mistakes and style, and model licences sometimes forbid it.

With access to the teacher's logits, the choice of divergence matters. **Forward KL** $\mathrm{KL}(p\|q)$ weights errors by the teacher's probabilities, so the student must put mass everywhere the teacher does. It is **mode-covering**. A low-capacity student that cannot match a multi-modal teacher spreads its mass, which leaves probability on text the teacher would never write. **Reverse KL** $\mathrm{KL}(q\|p)$ weights errors by the student's own probabilities. It is **mode-seeking**: the student may ignore some teacher modes but is punished for producing what the teacher finds unlikely.

MiniLLM (Gu, Dong, Wei and Huang, ICLR 2024) minimises reverse KL at the sequence level with a policy-gradient method and reports more precise, better-calibrated generations than forward-KL KD. GKD, generalized knowledge distillation (Agarwal et al., ICLR 2024), tackles the **train–inference mismatch**. A student trained only on teacher text never sees its own mistakes, the same exposure bias as **Teacher forcing**. GKD samples outputs from the *student* and has the teacher score every token, with a generalised Jensen–Shannon divergence that interpolates between forward and reverse KL.

> Figure: One next-token prediction, "The cat sat on the ___", over 8 candidate tokens with fixed teacher logits v = (5, 3.4, 2.9, 2.6, 1.2, −0.5, −2, −3). The student agrees on "mat" but ranks "sofa" above "floor". At T = 1 the teacher puts 0.140 on "floor" against the student's 0.012, and KL(p‖q) = 0.4018. At the default T = 4 the teacher is much flatter (0.266 on "mat"; 0.734 of its mass, the dark knowledge, lies elsewhere). KL drops to 0.0719, but T²·KL = 1.1505, close to its large-T limit 1.2655, the centred logit-matching loss. The gradient norm is 0.0320 without the T² factor and 0.5117 with it. The lower-right plot shows the raw gradient collapsing from 0.220 at T = 1 to 0.0014 at T = 20, while the scaled one stays near the hard-label gradient (0.431). With α = 0.5 the loss is 0.5·1.151 + 0.5·0.431 = 0.791. The soft term raises "floor" (+0.336) because the teacher ranks it second. The hard label lowers it (−0.012), because one-hot targets treat every wrong token alike. Reverse KL at T = 1 is 0.3027. It puts its largest push on the student's own top token and barely raises "floor" (+0.033, against +0.128 under forward KL), which the student already treats as unlikely. Blending the student 50% toward the teacher cuts T²·KL at T = 4 to 0.2978.

## Worked example: four tokens at T = 3

Teacher logits $v=(4,2,1,-1)$ and student logits $z=(3,0,1.5,-0.5)$ for the tokens (mat, floor, sofa, idea). The hard label is "mat". Use $T=3$ and $\alpha=0.7$.

Soft and hard terms (computed in Python with numpy)

| token | p at T=1 | p at T=3 | q at T=3 | ∂(T²KL)/∂z = T(q − p) | ∂CE/∂z (T=1) |
| --- | --- | --- | --- | --- | --- |
| mat (label) | 0.8390 | 0.4831 | 0.4375 | −0.1367 | −0.2326 |
| floor | 0.1135 | 0.2480 | 0.1609 | −0.2612 | +0.0382 |
| sofa | 0.0418 | 0.1777 | 0.2653 | +0.2629 | +0.1712 |
| idea | 0.0057 | 0.0912 | 0.1362 | +0.1350 | +0.0232 |

**Assembling the loss**

1. Softening: at $T=1$ the teacher gives 0.161 to tokens other than "mat"; at $T=3$ it gives 0.517. The ratio floor/sofa falls from $e^{1}=2.718$ to $e^{1/3}=1.396$. The ranking is kept, but the small probabilities become large enough to train on.
2. Soft term: $\mathrm{KL}(p^{(3)}\|q^{(3)})=0.0473$, times $T^2=9$: $0.4256$. Its gradient $T(q-p)$ is largest on "floor" ($-0.2612$, so raise it) and "sofa" ($+0.2629$, so lower it): exactly the ranking error the student makes.
3. Hard term: $\mathrm{CE}=-\ln q^{(1)}_{mat}=0.2648$. Its gradient is $q^{(1)}-\text{onehot}$, which *lowers* "floor" ($+0.0382$) along with every other wrong token. The label cannot say that "floor" is a reasonable answer.
4. Total: $0.7\cdot0.4256+0.3\cdot0.2648=0.3773$, with gradient $(-0.1655,-0.1714,+0.2354,+0.1014)$. "floor" ends up being raised.
5. High-temperature check: $T^2\mathrm{KL}$ is $0.5336$ at $T=10$ and $0.5602$ at $T=100$, approaching $\tfrac1{2N}\sum((z-\bar z)-(v-\bar v))^2=0.5625$. The scaled gradient approaches $((z-\bar z)-(v-\bar v))/N=(-0.125,-0.375,0.25,0.25)$, which is logit matching.

```python
import torch, torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

def kd_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5, ignore_index=-100):
    """student/teacher_logits: (B, L, V) predicting labels (B, L); labels = -100 on prompt and pad."""
    mask = labels != ignore_index
    V = min(student_logits.size(-1), teacher_logits.size(-1))      # guard against padded vocabularies
    s, t = student_logits[mask][:, :V].float(), teacher_logits[mask][:, :V].float()
    soft = F.kl_div(F.log_softmax(s / T, -1), F.log_softmax(t / T, -1),
                    log_target=True, reduction="batchmean") * T * T  # T^2 · KL(p_T ‖ q_T) per token
    hard = F.cross_entropy(s, labels[mask])
    return alpha * soft + (1 - alpha) * hard

# check on the worked example: one position, vocabulary (mat, floor, sofa, idea), label "mat"
z = torch.tensor([[[3.0, 0.0, 1.5, -0.5]]]); v = torch.tensor([[[4.0, 2.0, 1.0, -1.0]]])
print(kd_loss(z, v, torch.tensor([[0]]), T=3.0, alpha=0.7))        # 0.3773

# distilling a Llama 3.1 8B teacher into Llama 3.2 1B (same 128,256-token vocabulary)
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
teacher = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct",
                                               torch_dtype=torch.bfloat16).eval().requires_grad_(False)
student = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", torch_dtype=torch.bfloat16)
opt = torch.optim.AdamW(student.parameters(), lr=1e-5)

def step(input_ids, attention_mask, labels):          # labels already shifted: labels[:, t] = input_ids[:, t+1]
    with torch.no_grad():
        t_logits = teacher(input_ids=input_ids, attention_mask=attention_mask).logits
    s_logits = student(input_ids=input_ids, attention_mask=attention_mask).logits
    loss = kd_loss(s_logits, t_logits, labels, T=2.0, alpha=0.5)
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

# sequence-level KD (Kim & Rush): let the teacher write the targets, then ordinary SFT on them
prompt = tok.apply_chat_template([{"role": "user", "content": "Explain KV caching in two sentences."}],
                                 add_generation_prompt=True, return_tensors="pt")
y_hat = teacher.generate(prompt, max_new_tokens=128, num_beams=4, do_sample=False)
# ...add (prompt, y_hat) to the SFT set and train the student with plain cross-entropy
```

Token-level KD for causal LMs in PyTorch (Hinton loss over every supervised position), and a check on the worked example (written for PyTorch 2; the numbers in comments were computed with numpy)

> **Forgetting T² or mismatched vocabularies:** Drop the $T^2$ and the soft term's gradient shrinks by $1/T^2$. At $T=4$ in the figure the soft gradient falls from 0.512 to 0.032, and the hard label silently takes over. Logit KD also assumes the student and teacher share a tokenizer and vocabulary order. Across model families (for example a Qwen teacher and a Llama student) the per-token KL is meaningless without a vocabulary mapping, so use sequence-level KD on generated text instead.

> **Soft targets carry more information per example:** A one-hot label carries at most $\log_2 N$ bits and says nothing about the wrong classes. A teacher distribution ranks every class. That is why Hinton et al.'s speech student trained on 3% of the data reached 57.0% with soft targets but overfit to 44.5% with hard labels. It is also why Gemma 2 and Llama 3.2 pre-trained their small models on teacher distributions over the vocabulary rather than on one-hot next tokens.

> **Which samples the loss sees:** Logit KD and sequence-level KD both train on *teacher* (or dataset) text, which is off-policy for the student. At inference the student conditions on its own tokens, and its errors compound. On-policy methods such as GKD score the student's own samples with the teacher, much as **RLHF** scores samples with a reward model, but with a dense per-token signal.

**Strengths**

- Small, fast students that keep most of the teacher's quality (DistilBERT: 40% smaller, 97% of GLUE)
- Soft targets regularise and carry more information per example than labels
- Sequence-level KD needs only the teacher's text, so it works through an API
- Combines with **Quantization** and **Pruning** (Llama 3.2 1B/3B were pruned, then distilled)

**Limitations**

- Logit KD needs the teacher's full output distribution at every position, which is expensive at LLM vocabulary sizes
- Requires a shared tokenizer for token-level KL
- The student inherits the teacher's errors and biases, and cannot exceed it on the distilled task
- Off-policy training leaves exposure bias; on-policy KD costs student sampling plus teacher scoring

- **Compressing encoders:** **DistilBERT** and TinyBERT for classification and retrieval at a fraction of BERT's latency
- **Small LLM pre-training:** Gemma 2 2B/9B and Llama 3.2 1B/3B trained on teacher logits
- **Reasoning transfer:** DeepSeek-R1-Distill: SFT on long reasoning traces from a large RL-trained model
- **Faster decoding:** Kim and Rush's students decode greedily, with no beam search; distilled draft models also serve **Speculative decoding**

**Question:** Why does the Hinton loss multiply the soft-target KL by T²?

- A. To make the KL divergence symmetric
- B. Because the gradient of the softened KL with respect to the student logits scales as 1/T², so without T² the soft term would fade as T grows
- C. To keep the teacher's probabilities summing to one
- D. Because the student uses T = 1 at inference

**Answer:** B. Because the gradient of the softened KL with respect to the student logits scales as 1/T², so without T² the soft term would fade as T grows — $\partial\mathrm{KL}/\partial z_i=(q_i-p_i)/T$, and $q-p$ itself shrinks like $1/T$ at high temperature, so the gradient falls like $1/T^2$. Multiplying by $T^2$ keeps its size roughly constant, so the soft/hard mix $\alpha$ means the same thing at any $T$.

**Question:** A 1B student is distilled from a 70B teacher whose next-token distributions are often multi-modal. Which objective tends to avoid giving probability to text the teacher would not produce?

- A. Forward KL(p‖q) on teacher-generated text
- B. Reverse KL(q‖p), as in MiniLLM
- C. Cross-entropy on one-hot labels only
- D. Any divergence, since they share the same minimiser for every student

**Answer:** B. Reverse KL(q‖p), as in MiniLLM — They share a minimiser only if the student can represent the teacher exactly. A small student cannot. Forward KL is mode-covering: it must put mass wherever $p$ does, so it smears probability between modes. Reverse KL weights errors by $q$ and heavily penalises mass where $p$ is small, so it is mode-seeking.

**Key points**

- Hinton loss: $\alpha T^2\,\mathrm{KL}(\mathrm{softmax}(v/T)\|\mathrm{softmax}(z/T))+(1-\alpha)\,\mathrm{CE}(y,\mathrm{softmax}(z))$; the gradient on $z_i$ is $T(q_i-p_i)$ after the $T^2$.
- Temperature exposes dark knowledge, the teacher's ranking of wrong answers. As $T\to\infty$, $T^2\cdot$KL becomes squared error on centred logits (Ba and Caruana's logit matching).
- Three things to imitate: output distributions (logit KD), hidden features (FitNets, DistilBERT, TinyBERT), or whole teacher sequences (Kim & Rush sequence-level KD, i.e. SFT on teacher outputs).
- For LLMs: forward KL is mode-covering, reverse KL (MiniLLM) is mode-seeking, and on-policy KD (GKD) trains on student samples to remove exposure bias.
- Practice: Gemma 2 and Llama 3.2 small models were pre-trained on teacher logits; DeepSeek-R1-Distill-Qwen-32B reached 72.6% on AIME 2024 from SFT on R1 samples alone.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
