Definition
Teacher forcing is the standard way to train a model that generates a sequence one element at a time: at every training step the model's input is the ground-truth previous output (the "teacher" signal), not the model's own prediction . The loss is the sum of per-step losses along the true sequence.
Any autoregressive model, an RNN language model, a Seq2Seq decoder, a GPT, predicts from what came before. During generation "what came before" is necessarily its own output. During training we have a choice. Feeding the model's own predictions makes each step depend on every earlier step's sampling decisions: the model is lost for a long time early on (its own outputs are garbage), training is serial, and the gradient has to pass through discrete choices. Feeding the true prefix instead turns sequence training into ordinary supervised classification problems that share parameters.
Williams and Zipser ("A Learning Algorithm for Continually Running Fully Recurrent Neural Networks", Neural Computation, 1989) described the technique for their real-time recurrent learning algorithm: during training, the actual output of a unit that has a target is replaced by the teacher value in all later computation. They called it teacher-forced learning and found it speeds up and stabilises training on tasks such as oscillators. Goodfellow, Bengio and Courville's Deep Learning (2016, §10.2.1) explain it as what the maximum-likelihood criterion produces for models with output-to-hidden recurrence. The term exposure bias for its downside was coined by Ranzato, Chopra, Auli and Zaremba ("Sequence Level Training with Recurrent Neural Networks", ICLR 2016).
The objective
- the reference sequence; is a start token such as
<sos> - optional conditioning input (the source sentence in Seq2Seq, empty for a language model)
- the model's next-token distribution, computed from the true prefix
- by the chain rule the sum is exactly the sequence negative log-likelihood, so teacher forcing is maximum likelihood
The gradient is the sum of per-step cross-entropy gradients, . No gradient flows through the choice of the inputs, because the inputs are data. In an RNN the hidden state still links the steps, so the gradient still travels back through time in (BPTT), but never through a sampled token. Free-running training, by contrast, would need to differentiate through or through sampling, which is not differentiable without tricks such as REINFORCE or Gumbel-softmax.
At a glance
- Training input at step t
- ground truth
- Inference input at step t
- the model's own (greedy, sampled, or from a beam)
- Objective
- exact sequence log-likelihood (MLE)
- Parallelism
- all positions in one pass for Transformers and CNN decoders; per-step for RNNs, but no sequential sampling
- Main flaw
- exposure bias: train and test input distributions differ, so errors compound at generation
- Remedies
- scheduled sampling, professor forcing, sequence-level training (MIXER, minimum risk), noise in the inputs
Why Transformers depend on it
With teacher forcing the whole input sequence is known before the forward pass. A Transformer decoder with a causal mask (position may attend only to positions ) therefore computes all next-token distributions in one parallel pass, and the loss on all tokens comes from one matrix multiplication per layer. This is what makes training GPT-style models on trillions of tokens feasible. Generation is still sequential, one token per forward pass (with a key–value cache), because there the next input is not known until it has been produced. An RNN decoder must step through time even under teacher forcing, but it avoids the much worse serial dependence on its own sampled outputs. See Self-Attention.
Exposure bias: the train–test mismatch
Under teacher forcing the model only ever sees prefixes drawn from the data distribution. At generation it sees prefixes drawn from its own distribution, which contain its own mistakes. A small error at step moves the input at off the training distribution, where the model is less accurate, so the next error is more likely, and so on. The teacher-forced loss measures one-step-ahead accuracy and says little about how well the model does when it is left alone.
Three lessons sit in the figure. The training loss (blue) is an optimistic proxy for generation quality: two models whose teacher-forced losses differ by can differ by when free-running. Errors compound only in free-running mode, because only there are mistakes fed back. And an exact one-step model is not enough: with , the recursion is marginally stable (its roots lie on the unit circle), so a single wrong input never dies out. In discrete-token models the same mechanism produces repetitive loops and drift away from the prompt in long generations.
How much exposure bias hurts large modern language models is debated: they are trained on so much data that their own outputs are rarely far from the training distribution, and decoding methods (beam search, nucleus sampling) keep them on likely paths. For small models, long horizons, continuous outputs (time series, motion, audio) and closed-loop control, the effect is large and well documented.
Worked example: AR(2) on the figure's data
At , the teacher-forced least-squares fit gives (intercept below ), with training MSE 0.069. The last two training values are , . Both runs make the same first prediction, , because both are fed the same true history. From then on they diverge:
| h | observed y | teacher-forced ŷ (inputs: true y) | free-running ŷ (inputs: own ŷ) |
|---|---|---|---|
| 1 | 0.251 | 0.266 | 0.266 |
| 2 | 0.138 | 0.212 | 0.235 |
| 3 | 0.179 | 0.038 | 0.174 |
| 4 | 0.125 | 0.176 | 0.103 |
| 5 | 0.287 | 0.068 | 0.037 |
| 20 | −1.034 | −0.955 | 0.002 |
| 50 | −0.871 | −1.018 | 0.000 |
Step 2 of the free run uses its own : . The teacher-forced step uses the observed instead: . By the teacher-forced prediction still tracks the signal ( vs ) but the free run has collapsed to 0.002: its recursion has complex roots of modulus , so it decays like . Over the 100 test steps the teacher-forced MSE is 0.077 and the free-running MSE 0.690.
1import torch, torch.nn as nn, torch.nn.functional as F23V, H = 1000, 2564emb, cell, head = nn.Embedding(V, H), nn.GRUCell(H, H), nn.Linear(H, V)56def rnn_loss(y, h, teacher_forcing=True):7 """y: (N, T) with y[:, 0] = <sos>. Returns the mean token cross-entropy."""8 inp, loss = y[:, 0], 0.09 for t in range(1, y.size(1)):10 h = cell(emb(inp), h)11 logits = head(h)12 loss = loss + F.cross_entropy(logits, y[:, t])13 # teacher forcing: next input is the TRUE token; free running: the model's own guess14 inp = y[:, t] if teacher_forcing else logits.argmax(-1)15 return loss / (y.size(1) - 1)1617y = torch.randint(0, V, (8, 20)); h0 = torch.zeros(8, H)18print(rnn_loss(y, h0), rnn_loss(y, h0, teacher_forcing=False))1920# Transformer decoder: with teacher forcing all positions are trained in ONE pass21layer = nn.TransformerEncoderLayer(d_model=H, nhead=4, batch_first=True)22decoder = nn.TransformerEncoder(layer, num_layers=2) # decoder-only = causal encoder stack23x_in, labels = y[:, :-1], y[:, 1:] # shift by one24mask = nn.Transformer.generate_square_subsequent_mask(x_in.size(1))25logits = head(decoder(emb(x_in), mask=mask, is_causal=True)) # (8, 19, V)26loss = F.cross_entropy(logits.reshape(-1, V), labels.reshape(-1))2728# Hugging Face: passing labels= does the shift for you29# - causal LMs (GPT-2, Llama): model(input_ids=ids, labels=ids) shifts labels internally30# - encoder-decoders (T5, BART): model(input_ids=src, labels=tgt) builds31# decoder_input_ids by shifting tgt right and inserting the decoder start tokenUse when
- +Exactly maximum likelihood: a clean, well-understood objective
- +Fast, stable convergence: the model gets correct context from the first step
- +Every position can be trained in parallel in Transformers and CNN decoders
- +No gradients through discrete sampling decisions
Avoid when
- −Exposure bias: the model never sees its own mistakes during training
- −Teacher-forced loss can be far more optimistic than generation quality
- −Errors compound over long free-running horizons
- −Trains token-level likelihood, not the sequence-level metric you care about
Check yourself
A GPT-style model trains on a 2048-token sequence. How many forward passes does teacher forcing need to compute the loss on all 2047 next-token predictions?
Check yourself
Model A has a lower teacher-forced validation loss than model B, but B produces better long generations. What is the most likely explanation?
Remember
- ✓Teacher forcing feeds the true as input at step ; the loss is exactly the sequence NLL.
- ✓Introduced as teacher-forced learning by Williams & Zipser (1989); it is the default for RNNs, Seq2Seq and Transformers.
- ✓With a causal mask, Transformers compute all positions' losses in a single parallel pass.
- ✓Exposure bias (Ranzato et al., 2016): at generation the model consumes its own outputs, so errors compound.
- ✓Judge models by free-running metrics; remedies include scheduled sampling and professor forcing (Lamb et al., 2016).