Scheduled sampling

During training, feed the decoder its own sampled token instead of the ground truth with a probability that grows over training. A per-token coin flip with a decaying ε_i narrows the train/inference gap, but the objective is not consistent.

11 min read1277 wordsadvanced0 viewsupdated 2026-10-12

Definition

Scheduled sampling is a curriculum for training autoregressive decoders (Bengio, Vinyals, Jaitly and Shazeer, 2015). At every decoder step tt of mini-batch ii a coin is flipped: with probability εi\varepsilon_i the input is the true previous token yt−1y_{t-1} (teacher forcing), otherwise it is a token y^t−1\hat y_{t-1} drawn from the model's own prediction. εi\varepsilon_i starts near 1 and decays toward 0 on a fixed schedule, so training slides from teacher forcing toward the conditions the model will meet at inference.

A decoder trained with Teacher forcing only ever sees gold prefixes. At inference it must consume its own outputs, and one early mistake puts it in a state it never visited during training, where it may keep making mistakes. Ranzato et al. (2016) later named this train/test mismatch exposure bias. Scheduled sampling is the simplest fix: expose the model to its own mistakes during training, but only after it is good enough for its samples to be informative.

The idea comes from imitation learning, where SEARN (Daumé, Langford and Marcu, 2009) and DAgger (Ross, Gordon and Bagnell, 2011) train a policy on the states its own actions lead to. Bengio et al. made it online and cheap: one model, one training run, no extra passes over the data. In the paper's MSCOCO captioning experiments (Table 1) a 1-layer, 512-unit LSTM went from BLEU-4 28.8 to 30.6 and CIDEr 89.5 to 92.1, and the authors report that their scheduled-sampling entry ranked first on the final leaderboard of the 2015 MSCOCO captioning challenge.

The coin flip and the schedules

xt={yt−1with probability εiy^t−1∼Pθ( ⋅∣ht−1)with probability 1−εiLi=−∑t=1Tlog⁡Pθ(yt∣ht),ht=f(ht−1,xt)x_t=\begin{cases}y_{t-1} & \text{with probability }\varepsilon_i\\[2pt] \hat y_{t-1}\sim P_\theta(\,\cdot\mid h_{t-1}) & \text{with probability }1-\varepsilon_i\end{cases}\qquad \mathcal L_i=-\sum_{t=1}^{T}\log P_\theta(y_t\mid h_t),\quad h_t=f(h_{t-1},x_t)
ii
the mini-batch (training step) counter; the schedule is a function of training progress, not of position tt
εi\varepsilon_i
probability of feeding the ground truth. εi=1\varepsilon_i=1 is teacher forcing, εi=0\varepsilon_i=0 is free running
y^t−1\hat y_{t-1}
the model's own token: a sample from its softmax (the paper also allows the argmax)
yty_t
the target is always the gold token, whatever was fed as input
linear: εi=max⁡(ϵ,  k−ci)exponential: εi=k i  (k<1)inverse sigmoid: εi=kk+exp⁡(i/k)  (k≥1)\text{linear: }\varepsilon_i=\max(\epsilon,\;k-ci)\qquad \text{exponential: }\varepsilon_i=k^{\,i}\;(k<1)\qquad \text{inverse sigmoid: }\varepsilon_i=\frac{k}{k+\exp(i/k)}\;(k\ge1)
ϵ\epsilon
linear floor: the minimum amount of truth the model keeps receiving
k, ck,\,c
offset and slope (linear), base (exponential), or speed (inverse sigmoid); all set by how fast the model is expected to converge
i1/2i_{1/2}
the step where ε=0.5\varepsilon=0.5: (k−0.5)/c(k-0.5)/c, ln⁡0.5/ln⁡k\ln 0.5/\ln k and kln⁡kk\ln k respectively

Two details from the paper matter in practice. The coin is flipped for every token: flipping once per sequence gave results "as poor as" always sampling, because a fully free-running sequence early in training is mostly noise and its errors compound. And the sampling decision is not differentiated: the gradient of the loss at later steps does not flow back through which token was picked, so each step's loss is an ordinary cross-entropy on an input that happens to be noisy.

At a glance

Paper
Bengio, Vinyals, Jaitly, Shazeer, NeurIPS 2015
Fixes
exposure bias: training on gold prefixes, testing on model prefixes
Knob
schedule εi\varepsilon_i over mini-batches; the paper used inverse sigmoid for captioning and parsing, linear ramps for speech
Granularity
one coin per token (per sequence was much worse)
Model token
sampled from Pθ(⋅∣ht−1)P_\theta(\cdot\mid h_{t-1}), or its argmax; no gradient through the choice
Cost
no extra passes for an RNN, but the decoder can no longer be run in parallel over time
Caveat
improper objective (Huszár, 2015): its optimum need not be the data distribution
εᵢ = P(feed the true y₍t−1₎) at mini-batch i00.250.50.75102505007501000mini-batch ischeduleinverse sigmoidformulaεᵢ = k / (k + exp(i/k))k80iteration i250εᵢ0.7785ε = 0.5 at i350.6E[model inputs]2.21 / 10E[wrong inputs]0.66P(all gold)0.0818model share so far0.076decoder inputs at i = 250 (one coin per token): gold 8 · model 3 · wrong 1input<s>startat=1au=0.00dogt=2dogu=0.66runst=3runsu=0.20acrosst=4acrossu=0.54thet=5theu=0.07wett=6wetu=0.00grasst=7grassu=0.92towardt=8towardu=0.97itst=9itsu=0.21ownert=10!u=0.93.t=11targetgold if u < εᵢ, else the model's sample ŷ₍t−1₎ (stand-in model: right with prob. 0.70).The loss always compares against the gold target row; no gradient flows through the sampling.
fig.Top left: the three schedules over 1000 mini-batches (inverse sigmoid with k = 80 highlighted, exponential with k = 0.995, linear with c = 0.001 and floor 0.05); the dashed tick marks where the selected schedule crosses ε = 0.5 (i = 350.6, 138.3 and 500 respectively), and the readout on the right updates live. Below: the 11 decoder inputs of one caption at the chosen iteration. Input 1 is always <s>; the other 10 each flip their own coin u and take the gold token if u < εᵢ, otherwise the stand-in model's sample (orange if it happens to be right, red if wrong). At the default i = 250, ε = 0.7785, the expected number of model inputs is 2.21 of 10 and this draw has 3, one of them wrong; P(whole sequence teacher-forced) = ε¹⁰ = 0.0818. Drag i to 500 (ε = 0.1338) and most inputs are the model's. Toggle 'coin per sequence' to see the all-or-nothing variant the paper rejected.

How the schedules differ

The schedule decides when the model starts seeing its own tokens, and the three shapes spend the budget very differently. Exponential decay drops fast at the start, before the model's samples are any good. Linear decay is steady. The inverse sigmoid stays near 1 for a long warm-up, then falls steeply around i=kln⁡ki=k\ln k and flattens near 0. That is why it was the paper's choice for captioning and parsing: almost pure teacher forcing while the model learns the basics, then a quick switch.

schedulei = 0i = 250i = 500i = 750i = 1000ε = 0.5 at
linear, c=0.001c=0.001, floor 0.051.00000.75000.50000.25000.0500500.0
exponential, k=0.995k=0.9951.00000.28560.08160.02330.0067138.3
inverse sigmoid, k=80k=800.98770.77850.13380.00670.000380ln⁡80=350.680\ln 80=350.6
εᵢ for the figure's default schedules (computed in Node from the figure's lib)

The constants have to be scaled to the real length of training. For an inverse sigmoid to cross 0.5 at step 50 000 you need kln⁡k=50 000k\ln k=50\,000, i.e. k≈5 773k\approx5\,773. Averaged over the first 1000 steps, the default inverse sigmoid feeds model tokens at 64.8% of the inputs (the 'model share so far' readout), the exponential at 80.2% and the linear at 49.9%.

Why it helps, and why it is not a proper objective

When the input is a model sample, the target is still the gold token. The model is therefore trained to map slightly wrong prefixes back onto the correct continuation, which is exactly the recovery behaviour it needs at inference. The paper's speech experiment shows the trade-off cleanly (Table 3, TIMIT frame error rate): the teacher-forced baseline has the best next-step error with gold history (15.0%) but the worst decoding error (46.0%), while linear schedules from 0.25 or 0.5 down to 0 reach 34.5% and 35.0% when decoding.

Huszár (2015) showed the price: the scheduled-sampling objective is improper, so even with infinite data and capacity its minimiser is generally not the data distribution, and the learning algorithm is inconsistent. The reason is visible in the loss. When the input y^t−1\hat y_{t-1} was sampled independently of the gold prefix, the target yty_t carries information about the gold token that the input does not, and the best the model can do is predict a blend of the true conditional and the marginal.

Worked example: a two-token copy task (checked by Monte Carlo in Node)

  1. 1Data: y1∈{A,B}y_1\in\{A,B\} with probability 12\tfrac12 each, and y2=y1y_2=y_1. Only AAAA and BBBB ever occur. Step 1 is always fed <s>, so the model learns q(y1)=12q(y_1)=\tfrac12 exactly.
  2. 2Step 2 with input aa: with probability ε\varepsilon the input is the gold y1y_1, so the target is aa. With probability 1−ε1-\varepsilon it is the model's sample, independent of the gold y1y_1, so the target equals aa only half the time.
  3. 3The cross-entropy minimiser is the conditional frequency: q(y2=a∣a)=ε⋅12+(1−ε)⋅12⋅1212=1+ε2q(y_2=a\mid a)=\dfrac{\varepsilon\cdot\tfrac12+(1-\varepsilon)\cdot\tfrac12\cdot\tfrac12}{\tfrac12}=\dfrac{1+\varepsilon}{2}.
  4. 4At the end of training: ε=1\varepsilon=1 gives 1 (correct); ε=0.5\varepsilon=0.5 gives 0.75; ε=0\varepsilon=0 gives 0.5. A simulation with 10610^6 sequences gives 0.7502 and 0.4999.
  5. 5Free-running, the ε=0\varepsilon=0 model generates the impossible sequences ABAB or BABA with probability (1−ε)/2=0.5(1-\varepsilon)/2=0.5. Driving ε\varepsilon all the way to 0 has taught it to ignore its own history.
python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3def inverse_sigmoid(i, k): # eps_i = k / (k + exp(i/k)); 0.5 at i = k ln k
4 return k / (k + math.exp(min(i / k, 50.0)))
5
6class Decoder(nn.Module):
7 def __init__(self, vocab, emb=256, hid=512):
8 super().__init__()
9 self.emb = nn.Embedding(vocab, emb)
10 self.cell = nn.GRUCell(emb, hid)
11 self.out = nn.Linear(hid, vocab)
12
13 def forward(self, h, y, eps, sample=True):
14 # h: (N, hid) initial state from the encoder; y: (N, T) gold tokens, y[:, 0] = <sos>
15 N, T = y.shape
16 x = y[:, 0]
17 logits = []
18 for t in range(1, T):
19 h = self.cell(self.emb(x), h)
20 z = self.out(h) # (N, vocab)
21 logits.append(z)
22 with torch.no_grad(): # no gradient through the choice
23 pred = (torch.multinomial(F.softmax(z, -1), 1).squeeze(1)
24 if sample else z.argmax(-1))
25 gold = torch.rand(N, device=y.device) < eps # one coin per token
26 x = torch.where(gold, y[:, t], pred)
27 return torch.stack(logits, 1) # (N, T-1, vocab)
28
29# for i, (src, y) in enumerate(loader): # i counts mini-batches
30# eps = inverse_sigmoid(i, k=5773.0) # eps = 0.5 near step 50k
31# logits = dec(enc(src), y, eps)
32# loss = F.cross_entropy(logits.flatten(0, 1), y[:, 1:].flatten(), ignore_index=PAD)
A GRU decoder with per-token scheduled sampling and an inverse-sigmoid schedule

Transformers: parallel scheduled sampling

For an RNN the coin flip costs nothing, because the decoder runs step by step anyway. A Transformer decoder trained with teacher forcing processes all positions in one parallel pass, and true scheduled sampling would force it back into a sequential loop. Parallel scheduled sampling (Duckworth, Neelakantan, Goodrich, Kaiser and Bengio, 2019) keeps the parallelism: run a teacher-forced pass, sample a token at every position from its predictions, replace each gold input by that sample with probability 1−εi1-\varepsilon_i, and run another parallel pass on the mixed sequence, computing the loss against the gold targets. The mixed tokens are conditioned on the gold prefix rather than on the model's own earlier samples, which is the approximation that buys parallelism. The paper reports equal or better results than teacher forcing on image generation, summarisation, dialog and translation; Mihaylova and Martins (2019) proposed a similar two-pass decoder for Transformer NMT.

Related fixes attack exposure bias differently: professor forcing (Lamb et al., 2016) adds a discriminator that makes free-running hidden-state dynamics match teacher-forced ones; sequence-level training such as MIXER (Ranzato et al., 2016) optimises a test metric with REINFORCE on the model's own samples; Zhang et al. (2019) sample whole oracle sentences instead of single words.

Use when

  • +Directly targets exposure bias with a two-line change to the decoder loop
  • +No extra model, discriminator or reward; same cost per step for RNNs
  • +Gave consistent gains on captioning, parsing and speech in the original paper
  • +Robust per-token variant; a floor on ε keeps it safe

Avoid when

  • −Improper objective: its optimum is not the data distribution (Huszár, 2015)
  • −Breaks the time-parallel training of Transformers unless approximated (parallel scheduled sampling)
  • −Schedule constants depend on training length and need tuning
  • −No gradient through the sampled tokens, so it cannot learn which errors cause later errors

Check yourself

With the inverse-sigmoid schedule εᵢ = k/(k + exp(i/k)) and k = 80, at which mini-batch does the model first see its own token as often as the gold one?

Check yourself

In the two-token copy task (y₂ = y₁, both values equally likely), training ends with ε = 0. How often does the free-running model output a sequence that never occurs in the data?

Remember

  • ✓Per token, feed the gold yt−1y_{t-1} with probability εi\varepsilon_i, else the model's own sample; the target stays the gold yty_t.
  • ✓εi\varepsilon_i decays over mini-batches: linear max⁡(ϵ,k−ci)\max(\epsilon,k-ci), exponential kik^i, inverse sigmoid k/(k+ei/k)k/(k+e^{i/k}) with midpoint kln⁡kk\ln k.
  • ✓Flip the coin per token, not per sequence; do not back-propagate through the sampled choice.
  • ✓It narrows the train/inference gap (MSCOCO BLEU-4 28.8 → 30.6) but the objective is improper, so keep ε above 0.
  • ✓Transformers need parallel scheduled sampling: sample from a teacher-forced pass, mix, and run a second parallel pass.
← back to Deep Learning roadmap