# Teacher forcing

> Train an autoregressive model by feeding it the ground-truth previous token y_{t−1}, not its own prediction. It is exactly maximum likelihood and makes every position trainable in parallel, but at generation time the model must consume its own outputs, which it never saw in training.

Source: https://www.mlfoundry.in/learn/teacher-forcing  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 11 min read · updated 2026-10-12

**Teacher forcing:** is the standard way to train a model that generates a sequence one element at a time: at every training step $t$ the model's input is the **ground-truth** previous output $y_{t-1}$ (the "teacher" signal), not the model's own prediction $\hat y_{t-1}$. The loss is the sum of per-step losses $-\log p_\theta(y_t\mid y_{<t},x)$ along the true sequence.

Any autoregressive model, an RNN language model, a Seq2Seq decoder, a GPT, predicts $y_t$ from what came before. During generation "what came before" is necessarily its own output. During training we have a choice. Feeding the model's own predictions makes each step depend on every earlier step's sampling decisions: the model is lost for a long time early on (its own outputs are garbage), training is serial, and the gradient has to pass through discrete choices. Feeding the true prefix instead turns sequence training into $T$ ordinary supervised classification problems that share parameters.

Williams and Zipser ("A Learning Algorithm for Continually Running Fully Recurrent Neural Networks", Neural Computation, 1989) described the technique for their real-time recurrent learning algorithm: during training, the actual output of a unit that has a target is replaced by the teacher value in all later computation. They called it **teacher-forced** learning and found it speeds up and stabilises training on tasks such as oscillators. Goodfellow, Bengio and Courville's *Deep Learning* (2016, §10.2.1) explain it as what the maximum-likelihood criterion produces for models with output-to-hidden recurrence. The term **exposure bias** for its downside was coined by Ranzato, Chopra, Auli and Zaremba ("Sequence Level Training with Recurrent Neural Networks", ICLR 2016).

## The objective

$$
\mathcal L_{\text{TF}}(\theta)=-\sum_{t=1}^{T}\log p_\theta\big(y_t\mid \underbrace{y_1,\dots,y_{t-1}}_{\text{ground truth}},\,x\big)\;=\;-\log p_\theta(y\mid x)
$$

- $y_1..y_T$: the reference sequence; $y_0$ is a start token such as `<sos>`
- $x$: optional conditioning input (the source sentence in Seq2Seq, empty for a language model)
- $p_\theta(y_t\mid y_{<t},x)$: the model's next-token distribution, computed from the **true** prefix
- $-\log p_\theta(y\mid x)$: by the chain rule the sum is exactly the sequence negative log-likelihood, so teacher forcing **is** maximum likelihood

The gradient is the sum of per-step cross-entropy gradients, $\nabla_\theta\mathcal L=-\sum_t\nabla_\theta\log p_\theta(y_t\mid y_{<t},x)$. No gradient flows through the choice of the inputs, because the inputs are data. In an RNN the hidden state still links the steps, so the gradient still travels back through time in $h_t$ (BPTT), but never through a sampled token. Free-running training, by contrast, would need to differentiate through $\hat y_{t-1}=\arg\max$ or through sampling, which is not differentiable without tricks such as REINFORCE or Gumbel-softmax.

**At a glance**

- **Training input at step t:** ground truth $y_{t-1}$
- **Inference input at step t:** the model's own $\hat y_{t-1}$ (greedy, sampled, or from a beam)
- **Objective:** exact sequence log-likelihood (MLE)
- **Parallelism:** all $T$ positions in one pass for Transformers and CNN decoders; per-step for RNNs, but no sequential sampling
- **Main flaw:** **exposure bias**: train and test input distributions differ, so errors compound at generation
- **Remedies:** scheduled sampling, professor forcing, sequence-level training (MIXER, minimum risk), noise in the inputs

## Why Transformers depend on it

With teacher forcing the whole input sequence $[y_0,y_1,\dots,y_{T-1}]$ is known before the forward pass. A Transformer decoder with a causal mask (position $t$ may attend only to positions $\le t$) therefore computes all $T$ next-token distributions **in one parallel pass**, and the loss on all $T$ tokens comes from one matrix multiplication per layer. This is what makes training GPT-style models on trillions of tokens feasible. Generation is still sequential, one token per forward pass (with a key–value cache), because there the next input is not known until it has been produced. An RNN decoder must step through time even under teacher forcing, but it avoids the much worse serial dependence on its own sampled outputs. See **Self-Attention**.

## Exposure bias: the train–test mismatch

Under teacher forcing the model only ever sees prefixes drawn from the data distribution. At generation it sees prefixes drawn from **its own** distribution, which contain its own mistakes. A small error at step $t$ moves the input at $t+1$ off the training distribution, where the model is less accurate, so the next error is more likely, and so on. The teacher-forced loss measures one-step-ahead accuracy and says little about how well the model does when it is left alone.

> Figure: An autoregressive model ŷₜ = c + Σₖ aₖ·yₜ₋ₖ is fitted with teacher forcing (least squares on the true history) to 200 noisy samples of a sum of two sinusoids, then run on the next 100 steps two ways: teacher-forced (blue, fed the true previous values) and free-running (orange, fed its own predictions). At the default p = 4, σ = 0.1 the teacher-forced test MSE is 0.072 but the free-running MSE is 0.695, 9.7× larger: the fitted recursion has spectral radius 0.813, so the free run dies out to the mean, and its error approaches the signal's variance of 0.68. Raise p to 10 and the training loss falls from 0.063 to 0.022 (2.9×), but the free-running error falls from 0.695 to 0.046. With σ = 0 and p = 4 the fit is exact, yet one injected mistake of +1 sends the free run into oscillations with squared errors above 100, while the teacher-forced run absorbs it in a single step.

Three lessons sit in the figure. The training loss (blue) is an optimistic proxy for generation quality: two models whose teacher-forced losses differ by $2\times$ can differ by $15\times$ when free-running. Errors **compound** only in free-running mode, because only there are mistakes fed back. And an exact one-step model is not enough: with $\sigma=0$, $p=4$ the recursion is marginally stable (its roots lie on the unit circle), so a single wrong input never dies out. In discrete-token models the same mechanism produces repetitive loops and drift away from the prompt in long generations.

How much exposure bias hurts large modern language models is debated: they are trained on so much data that their own outputs are rarely far from the training distribution, and decoding methods (beam search, nucleus sampling) keep them on likely paths. For small models, long horizons, continuous outputs (time series, motion, audio) and closed-loop control, the effect is large and well documented.

## Worked example: AR(2) on the figure's data

At $\sigma=0.1$, $p=2$ the teacher-forced least-squares fit gives $\hat y_t=1.508\,y_{t-1}-0.678\,y_{t-2}$ (intercept below $10^{-4}$), with training MSE 0.069. The last two training values are $y_{198}=0.152$, $y_{199}=0.245$. Both runs make the same first prediction, $1.508\cdot0.245-0.678\cdot0.152=0.266$, because both are fed the same true history. From then on they diverge:

First test steps of the AR(2) model (σ = 0.1), from the figure's library

| h | observed y | teacher-forced ŷ (inputs: true y) | free-running ŷ (inputs: own ŷ) |
| --- | --- | --- | --- |
| 1 | 0.251 | 0.266 | 0.266 |
| 2 | 0.138 | 0.212 | 0.235 |
| 3 | 0.179 | 0.038 | 0.174 |
| 4 | 0.125 | 0.176 | 0.103 |
| 5 | 0.287 | 0.068 | 0.037 |
| 20 | −1.034 | −0.955 | 0.002 |
| 50 | −0.871 | −1.018 | 0.000 |

Step 2 of the free run uses its own $\hat y_{200}=0.266$: $1.508\cdot0.266-0.678\cdot0.245=0.235$. The teacher-forced step uses the observed $y_{200}=0.251$ instead: $1.508\cdot0.251-0.678\cdot0.245=0.212$. By $h=20$ the teacher-forced prediction still tracks the signal ($-0.955$ vs $-1.034$) but the free run has collapsed to 0.002: its recursion has complex roots of modulus $\sqrt{0.678}=0.82$, so it decays like $0.82^h$. Over the 100 test steps the teacher-forced MSE is 0.077 and the free-running MSE 0.690.

```python
import torch, torch.nn as nn, torch.nn.functional as F

V, H = 1000, 256
emb, cell, head = nn.Embedding(V, H), nn.GRUCell(H, H), nn.Linear(H, V)

def rnn_loss(y, h, teacher_forcing=True):
    """y: (N, T) with y[:, 0] = <sos>. Returns the mean token cross-entropy."""
    inp, loss = y[:, 0], 0.0
    for t in range(1, y.size(1)):
        h = cell(emb(inp), h)
        logits = head(h)
        loss = loss + F.cross_entropy(logits, y[:, t])
        # teacher forcing: next input is the TRUE token; free running: the model's own guess
        inp = y[:, t] if teacher_forcing else logits.argmax(-1)
    return loss / (y.size(1) - 1)

y = torch.randint(0, V, (8, 20)); h0 = torch.zeros(8, H)
print(rnn_loss(y, h0), rnn_loss(y, h0, teacher_forcing=False))

# Transformer decoder: with teacher forcing all positions are trained in ONE pass
layer = nn.TransformerEncoderLayer(d_model=H, nhead=4, batch_first=True)
decoder = nn.TransformerEncoder(layer, num_layers=2)        # decoder-only = causal encoder stack
x_in, labels = y[:, :-1], y[:, 1:]                          # shift by one
mask = nn.Transformer.generate_square_subsequent_mask(x_in.size(1))
logits = head(decoder(emb(x_in), mask=mask, is_causal=True))   # (8, 19, V)
loss = F.cross_entropy(logits.reshape(-1, V), labels.reshape(-1))

# Hugging Face: passing labels= does the shift for you
# - causal LMs (GPT-2, Llama): model(input_ids=ids, labels=ids) shifts labels internally
# - encoder-decoders (T5, BART): model(input_ids=src, labels=tgt) builds
#   decoder_input_ids by shifting tgt right and inserting the decoder start token
```

Teacher forcing in an RNN decoder, in a Transformer, and in Hugging Face models

> **Off-by-one shifts:** The input at position $t$ must be $y_{t-1}$ and the label $y_t$. Feeding the unshifted sequence as both input and label lets a Transformer copy the label through attention to position $t$ itself (or an RNN through its input), so the loss drops to almost zero and the model learns nothing. A training loss that is suspiciously low from the first steps is the classic sign. Combine the shift with a causal mask; forgetting the mask is the same leak.

> **Monitor the free-running metric:** Because the teacher-forced loss can look excellent while generation fails, evaluate what you actually deploy: decode on a validation set (greedy or beam) and measure BLEU, exact match, rollout error at several horizons, or task success. In the figure the training loss changes $2.9\times$ between $p=4$ and $p=10$ while the free-running error changes $15\times$.

> **Remedies, briefly:** **Scheduled sampling** (Bengio et al., 2015) replaces some ground-truth inputs by model samples, more of them as training goes on (see **Scheduled sampling**). **Professor forcing** (Lamb, Goyal, Zhang, Zhang, Courville and Bengio, NeurIPS 2016) trains a discriminator to tell teacher-forced from free-running hidden-state trajectories and pushes the generator to make them indistinguishable. Sequence-level objectives such as MIXER (Ranzato et al., 2016) train on the model's own outputs with a reward like BLEU. For time series, adding noise to the teacher-forced inputs makes the model robust to its own errors.

**Strengths**

- Exactly maximum likelihood: a clean, well-understood objective
- Fast, stable convergence: the model gets correct context from the first step
- Every position can be trained in parallel in Transformers and CNN decoders
- No gradients through discrete sampling decisions

**Limitations**

- Exposure bias: the model never sees its own mistakes during training
- Teacher-forced loss can be far more optimistic than generation quality
- Errors compound over long free-running horizons
- Trains token-level likelihood, not the sequence-level metric you care about

**Question:** A GPT-style model trains on a 2048-token sequence. How many forward passes does teacher forcing need to compute the loss on all 2047 next-token predictions?

- A. One
- B. 2047, one per position
- C. log₂ 2048 = 11
- D. It depends on the batch size

**Answer:** A. One — The input is the known ground-truth sequence, so with a causal mask every position's prediction is computed in the same pass: position $t$ attends to tokens $\le t$ only. Generation, which must wait for each token, needs one pass per new token.

**Question:** Model A has a lower teacher-forced validation loss than model B, but B produces better long generations. What is the most likely explanation?

- A. Exposure bias: A's one-step accuracy does not carry over when it consumes its own outputs
- B. B was trained with a larger learning rate
- C. The validation loss was computed with teacher forcing turned off
- D. Teacher forcing is not maximum likelihood

**Answer:** A. Exposure bias: A's one-step accuracy does not carry over when it consumes its own outputs — Teacher-forced loss scores one-step-ahead predictions from true prefixes. Free-running generation compounds errors, and models differ in how fast they drift. In the figure an AR(4) and an AR(10) differ by $2\times$ in teacher-forced test MSE (0.072 vs 0.035) but by $15\times$ free-running (0.695 vs 0.046).

**Key points**

- Teacher forcing feeds the true $y_{t-1}$ as input at step $t$; the loss $-\sum_t\log p(y_t\mid y_{<t})$ is exactly the sequence NLL.
- Introduced as teacher-forced learning by Williams & Zipser (1989); it is the default for RNNs, Seq2Seq and Transformers.
- With a causal mask, Transformers compute all positions' losses in a single parallel pass.
- Exposure bias (Ranzato et al., 2016): at generation the model consumes its own outputs, so errors compound.
- Judge models by free-running metrics; remedies include scheduled sampling and professor forcing (Lamb et al., 2016).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
