Teacher forcing

Train an autoregressive model by feeding it the ground-truth previous token y_{t−1}, not its own prediction. It is exactly maximum likelihood and makes every position trainable in parallel, but at generation time the model must consume its own outputs, which it never saw in training.

11 min read1115 wordscore0 viewsupdated 2026-10-12

Definition

Teacher forcing is the standard way to train a model that generates a sequence one element at a time: at every training step tt the model's input is the ground-truth previous output yt−1y_{t-1} (the "teacher" signal), not the model's own prediction y^t−1\hat y_{t-1}. The loss is the sum of per-step losses −log⁡pθ(yt∣y<t,x)-\log p_\theta(y_t\mid y_{<t},x) along the true sequence.

Any autoregressive model, an RNN language model, a Seq2Seq decoder, a GPT, predicts yty_t from what came before. During generation "what came before" is necessarily its own output. During training we have a choice. Feeding the model's own predictions makes each step depend on every earlier step's sampling decisions: the model is lost for a long time early on (its own outputs are garbage), training is serial, and the gradient has to pass through discrete choices. Feeding the true prefix instead turns sequence training into TT ordinary supervised classification problems that share parameters.

Williams and Zipser ("A Learning Algorithm for Continually Running Fully Recurrent Neural Networks", Neural Computation, 1989) described the technique for their real-time recurrent learning algorithm: during training, the actual output of a unit that has a target is replaced by the teacher value in all later computation. They called it teacher-forced learning and found it speeds up and stabilises training on tasks such as oscillators. Goodfellow, Bengio and Courville's Deep Learning (2016, §10.2.1) explain it as what the maximum-likelihood criterion produces for models with output-to-hidden recurrence. The term exposure bias for its downside was coined by Ranzato, Chopra, Auli and Zaremba ("Sequence Level Training with Recurrent Neural Networks", ICLR 2016).

The objective

LTF(θ)=−∑t=1Tlog⁡pθ(yt∣y1,…,yt−1⏟ground truth, x)  =  −log⁡pθ(y∣x)\mathcal L_{\text{TF}}(\theta)=-\sum_{t=1}^{T}\log p_\theta\big(y_t\mid \underbrace{y_1,\dots,y_{t-1}}_{\text{ground truth}},\,x\big)\;=\;-\log p_\theta(y\mid x)
y1..yTy_1..y_T
the reference sequence; y0y_0 is a start token such as <sos>
xx
optional conditioning input (the source sentence in Seq2Seq, empty for a language model)
pθ(yt∣y<t,x)p_\theta(y_t\mid y_{<t},x)
the model's next-token distribution, computed from the true prefix
−log⁡pθ(y∣x)-\log p_\theta(y\mid x)
by the chain rule the sum is exactly the sequence negative log-likelihood, so teacher forcing is maximum likelihood

The gradient is the sum of per-step cross-entropy gradients, ∇θL=−∑t∇θlog⁡pθ(yt∣y<t,x)\nabla_\theta\mathcal L=-\sum_t\nabla_\theta\log p_\theta(y_t\mid y_{<t},x). No gradient flows through the choice of the inputs, because the inputs are data. In an RNN the hidden state still links the steps, so the gradient still travels back through time in hth_t (BPTT), but never through a sampled token. Free-running training, by contrast, would need to differentiate through y^t−1=arg⁡max⁡\hat y_{t-1}=\arg\max or through sampling, which is not differentiable without tricks such as REINFORCE or Gumbel-softmax.

At a glance

Training input at step t
ground truth yt−1y_{t-1}
Inference input at step t
the model's own y^t−1\hat y_{t-1} (greedy, sampled, or from a beam)
Objective
exact sequence log-likelihood (MLE)
Parallelism
all TT positions in one pass for Transformers and CNN decoders; per-step for RNNs, but no sequential sampling
Main flaw
exposure bias: train and test input distributions differ, so errors compound at generation
Remedies
scheduled sampling, professor forcing, sequence-level training (MIXER, minimum risk), noise in the inputs

Why Transformers depend on it

With teacher forcing the whole input sequence [y0,y1,…,yT−1][y_0,y_1,\dots,y_{T-1}] is known before the forward pass. A Transformer decoder with a causal mask (position tt may attend only to positions ≤t\le t) therefore computes all TT next-token distributions in one parallel pass, and the loss on all TT tokens comes from one matrix multiplication per layer. This is what makes training GPT-style models on trillions of tokens feasible. Generation is still sequential, one token per forward pass (with a key–value cache), because there the next input is not known until it has been produced. An RNN decoder must step through time even under teacher forcing, but it avoids the much worse serial dependence on its own sampled outputs. See Self-Attention.

Exposure bias: the train–test mismatch

Under teacher forcing the model only ever sees prefixes drawn from the data distribution. At generation it sees prefixes drawn from its own distribution, which contain its own mistakes. A small error at step tt moves the input at t+1t+1 off the training distribution, where the model is less accurate, so the next error is more likely, and so on. The teacher-forced loss measures one-step-ahead accuracy and says little about how well the model does when it is left alone.

ŷₜ = c + Σₖ aₖ·(input)ₜ₋ₖ, p = 4 · trained with teacher forcing on t < 200training datatest: predict t = 200…299test MSE over the first h steps (log)10.01h = 1h = 100training loss (teacher-forced)0.063test MSE, teacher-forced0.072test MSE, free-running0.695free-running / teacher-forced9.7×noise floor σ²0.010AR spectral radius ρ0.813ρ < 1: the free run decays toward the mean● observed yₜ ━ teacher-forced (true inputs) ━ free-running (own outputs fed back)
fig.An autoregressive model ŷₜ = c + Σₖ aₖ·yₜ₋ₖ is fitted with teacher forcing (least squares on the true history) to 200 noisy samples of a sum of two sinusoids, then run on the next 100 steps two ways: teacher-forced (blue, fed the true previous values) and free-running (orange, fed its own predictions). At the default p = 4, σ = 0.1 the teacher-forced test MSE is 0.072 but the free-running MSE is 0.695, 9.7× larger: the fitted recursion has spectral radius 0.813, so the free run dies out to the mean, and its error approaches the signal's variance of 0.68. Raise p to 10 and the training loss falls from 0.063 to 0.022 (2.9×), but the free-running error falls from 0.695 to 0.046. With σ = 0 and p = 4 the fit is exact, yet one injected mistake of +1 sends the free run into oscillations with squared errors above 100, while the teacher-forced run absorbs it in a single step.

Three lessons sit in the figure. The training loss (blue) is an optimistic proxy for generation quality: two models whose teacher-forced losses differ by 2×2\times can differ by 15×15\times when free-running. Errors compound only in free-running mode, because only there are mistakes fed back. And an exact one-step model is not enough: with σ=0\sigma=0, p=4p=4 the recursion is marginally stable (its roots lie on the unit circle), so a single wrong input never dies out. In discrete-token models the same mechanism produces repetitive loops and drift away from the prompt in long generations.

How much exposure bias hurts large modern language models is debated: they are trained on so much data that their own outputs are rarely far from the training distribution, and decoding methods (beam search, nucleus sampling) keep them on likely paths. For small models, long horizons, continuous outputs (time series, motion, audio) and closed-loop control, the effect is large and well documented.

Worked example: AR(2) on the figure's data

At σ=0.1\sigma=0.1, p=2p=2 the teacher-forced least-squares fit gives y^t=1.508 yt−1−0.678 yt−2\hat y_t=1.508\,y_{t-1}-0.678\,y_{t-2} (intercept below 10−410^{-4}), with training MSE 0.069. The last two training values are y198=0.152y_{198}=0.152, y199=0.245y_{199}=0.245. Both runs make the same first prediction, 1.508⋅0.245−0.678⋅0.152=0.2661.508\cdot0.245-0.678\cdot0.152=0.266, because both are fed the same true history. From then on they diverge:

hobserved yteacher-forced ŷ (inputs: true y)free-running ŷ (inputs: own ŷ)
10.2510.2660.266
20.1380.2120.235
30.1790.0380.174
40.1250.1760.103
50.2870.0680.037
20−1.034−0.9550.002
50−0.871−1.0180.000
First test steps of the AR(2) model (σ = 0.1), from the figure's library

Step 2 of the free run uses its own y^200=0.266\hat y_{200}=0.266: 1.508⋅0.266−0.678⋅0.245=0.2351.508\cdot0.266-0.678\cdot0.245=0.235. The teacher-forced step uses the observed y200=0.251y_{200}=0.251 instead: 1.508⋅0.251−0.678⋅0.245=0.2121.508\cdot0.251-0.678\cdot0.245=0.212. By h=20h=20 the teacher-forced prediction still tracks the signal (−0.955-0.955 vs −1.034-1.034) but the free run has collapsed to 0.002: its recursion has complex roots of modulus 0.678=0.82\sqrt{0.678}=0.82, so it decays like 0.82h0.82^h. Over the 100 test steps the teacher-forced MSE is 0.077 and the free-running MSE 0.690.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3V, H = 1000, 256
4emb, cell, head = nn.Embedding(V, H), nn.GRUCell(H, H), nn.Linear(H, V)
5
6def rnn_loss(y, h, teacher_forcing=True):
7 """y: (N, T) with y[:, 0] = <sos>. Returns the mean token cross-entropy."""
8 inp, loss = y[:, 0], 0.0
9 for t in range(1, y.size(1)):
10 h = cell(emb(inp), h)
11 logits = head(h)
12 loss = loss + F.cross_entropy(logits, y[:, t])
13 # teacher forcing: next input is the TRUE token; free running: the model's own guess
14 inp = y[:, t] if teacher_forcing else logits.argmax(-1)
15 return loss / (y.size(1) - 1)
16
17y = torch.randint(0, V, (8, 20)); h0 = torch.zeros(8, H)
18print(rnn_loss(y, h0), rnn_loss(y, h0, teacher_forcing=False))
19
20# Transformer decoder: with teacher forcing all positions are trained in ONE pass
21layer = nn.TransformerEncoderLayer(d_model=H, nhead=4, batch_first=True)
22decoder = nn.TransformerEncoder(layer, num_layers=2) # decoder-only = causal encoder stack
23x_in, labels = y[:, :-1], y[:, 1:] # shift by one
24mask = nn.Transformer.generate_square_subsequent_mask(x_in.size(1))
25logits = head(decoder(emb(x_in), mask=mask, is_causal=True)) # (8, 19, V)
26loss = F.cross_entropy(logits.reshape(-1, V), labels.reshape(-1))
27
28# Hugging Face: passing labels= does the shift for you
29# - causal LMs (GPT-2, Llama): model(input_ids=ids, labels=ids) shifts labels internally
30# - encoder-decoders (T5, BART): model(input_ids=src, labels=tgt) builds
31# decoder_input_ids by shifting tgt right and inserting the decoder start token
Teacher forcing in an RNN decoder, in a Transformer, and in Hugging Face models

Use when

  • +Exactly maximum likelihood: a clean, well-understood objective
  • +Fast, stable convergence: the model gets correct context from the first step
  • +Every position can be trained in parallel in Transformers and CNN decoders
  • +No gradients through discrete sampling decisions

Avoid when

  • −Exposure bias: the model never sees its own mistakes during training
  • −Teacher-forced loss can be far more optimistic than generation quality
  • −Errors compound over long free-running horizons
  • −Trains token-level likelihood, not the sequence-level metric you care about

Check yourself

A GPT-style model trains on a 2048-token sequence. How many forward passes does teacher forcing need to compute the loss on all 2047 next-token predictions?

Check yourself

Model A has a lower teacher-forced validation loss than model B, but B produces better long generations. What is the most likely explanation?

Remember

  • ✓Teacher forcing feeds the true yt−1y_{t-1} as input at step tt; the loss −∑tlog⁡p(yt∣y<t)-\sum_t\log p(y_t\mid y_{<t}) is exactly the sequence NLL.
  • ✓Introduced as teacher-forced learning by Williams & Zipser (1989); it is the default for RNNs, Seq2Seq and Transformers.
  • ✓With a causal mask, Transformers compute all positions' losses in a single parallel pass.
  • ✓Exposure bias (Ranzato et al., 2016): at generation the model consumes its own outputs, so errors compound.
  • ✓Judge models by free-running metrics; remedies include scheduled sampling and professor forcing (Lamb et al., 2016).
← back to Deep Learning roadmap