Definition
Scheduled sampling is a curriculum for training autoregressive decoders (Bengio, Vinyals, Jaitly and Shazeer, 2015). At every decoder step of mini-batch a coin is flipped: with probability the input is the true previous token (teacher forcing), otherwise it is a token drawn from the model's own prediction. starts near 1 and decays toward 0 on a fixed schedule, so training slides from teacher forcing toward the conditions the model will meet at inference.
A decoder trained with Teacher forcing only ever sees gold prefixes. At inference it must consume its own outputs, and one early mistake puts it in a state it never visited during training, where it may keep making mistakes. Ranzato et al. (2016) later named this train/test mismatch exposure bias. Scheduled sampling is the simplest fix: expose the model to its own mistakes during training, but only after it is good enough for its samples to be informative.
The idea comes from imitation learning, where SEARN (Daumé, Langford and Marcu, 2009) and DAgger (Ross, Gordon and Bagnell, 2011) train a policy on the states its own actions lead to. Bengio et al. made it online and cheap: one model, one training run, no extra passes over the data. In the paper's MSCOCO captioning experiments (Table 1) a 1-layer, 512-unit LSTM went from BLEU-4 28.8 to 30.6 and CIDEr 89.5 to 92.1, and the authors report that their scheduled-sampling entry ranked first on the final leaderboard of the 2015 MSCOCO captioning challenge.
The coin flip and the schedules
- the mini-batch (training step) counter; the schedule is a function of training progress, not of position
- probability of feeding the ground truth. is teacher forcing, is free running
- the model's own token: a sample from its softmax (the paper also allows the argmax)
- the target is always the gold token, whatever was fed as input
- linear floor: the minimum amount of truth the model keeps receiving
- offset and slope (linear), base (exponential), or speed (inverse sigmoid); all set by how fast the model is expected to converge
- the step where : , and respectively
Two details from the paper matter in practice. The coin is flipped for every token: flipping once per sequence gave results "as poor as" always sampling, because a fully free-running sequence early in training is mostly noise and its errors compound. And the sampling decision is not differentiated: the gradient of the loss at later steps does not flow back through which token was picked, so each step's loss is an ordinary cross-entropy on an input that happens to be noisy.
At a glance
- Paper
- Bengio, Vinyals, Jaitly, Shazeer, NeurIPS 2015
- Fixes
- exposure bias: training on gold prefixes, testing on model prefixes
- Knob
- schedule over mini-batches; the paper used inverse sigmoid for captioning and parsing, linear ramps for speech
- Granularity
- one coin per token (per sequence was much worse)
- Model token
- sampled from , or its argmax; no gradient through the choice
- Cost
- no extra passes for an RNN, but the decoder can no longer be run in parallel over time
- Caveat
- improper objective (Huszár, 2015): its optimum need not be the data distribution
How the schedules differ
The schedule decides when the model starts seeing its own tokens, and the three shapes spend the budget very differently. Exponential decay drops fast at the start, before the model's samples are any good. Linear decay is steady. The inverse sigmoid stays near 1 for a long warm-up, then falls steeply around and flattens near 0. That is why it was the paper's choice for captioning and parsing: almost pure teacher forcing while the model learns the basics, then a quick switch.
| schedule | i = 0 | i = 250 | i = 500 | i = 750 | i = 1000 | ε = 0.5 at |
|---|---|---|---|---|---|---|
| linear, , floor 0.05 | 1.0000 | 0.7500 | 0.5000 | 0.2500 | 0.0500 | 500.0 |
| exponential, | 1.0000 | 0.2856 | 0.0816 | 0.0233 | 0.0067 | 138.3 |
| inverse sigmoid, | 0.9877 | 0.7785 | 0.1338 | 0.0067 | 0.0003 |
The constants have to be scaled to the real length of training. For an inverse sigmoid to cross 0.5 at step 50 000 you need , i.e. . Averaged over the first 1000 steps, the default inverse sigmoid feeds model tokens at 64.8% of the inputs (the 'model share so far' readout), the exponential at 80.2% and the linear at 49.9%.
Why it helps, and why it is not a proper objective
When the input is a model sample, the target is still the gold token. The model is therefore trained to map slightly wrong prefixes back onto the correct continuation, which is exactly the recovery behaviour it needs at inference. The paper's speech experiment shows the trade-off cleanly (Table 3, TIMIT frame error rate): the teacher-forced baseline has the best next-step error with gold history (15.0%) but the worst decoding error (46.0%), while linear schedules from 0.25 or 0.5 down to 0 reach 34.5% and 35.0% when decoding.
Huszár (2015) showed the price: the scheduled-sampling objective is improper, so even with infinite data and capacity its minimiser is generally not the data distribution, and the learning algorithm is inconsistent. The reason is visible in the loss. When the input was sampled independently of the gold prefix, the target carries information about the gold token that the input does not, and the best the model can do is predict a blend of the true conditional and the marginal.
Worked example: a two-token copy task (checked by Monte Carlo in Node)
- 1Data: with probability each, and . Only and ever occur. Step 1 is always fed <s>, so the model learns exactly.
- 2Step 2 with input : with probability the input is the gold , so the target is . With probability it is the model's sample, independent of the gold , so the target equals only half the time.
- 3The cross-entropy minimiser is the conditional frequency: .
- 4At the end of training: gives 1 (correct); gives 0.75; gives 0.5. A simulation with sequences gives 0.7502 and 0.4999.
- 5Free-running, the model generates the impossible sequences or with probability . Driving all the way to 0 has taught it to ignore its own history.
1import math, torch, torch.nn as nn, torch.nn.functional as F23def inverse_sigmoid(i, k): # eps_i = k / (k + exp(i/k)); 0.5 at i = k ln k4 return k / (k + math.exp(min(i / k, 50.0)))56class Decoder(nn.Module):7 def __init__(self, vocab, emb=256, hid=512):8 super().__init__()9 self.emb = nn.Embedding(vocab, emb)10 self.cell = nn.GRUCell(emb, hid)11 self.out = nn.Linear(hid, vocab)1213 def forward(self, h, y, eps, sample=True):14 # h: (N, hid) initial state from the encoder; y: (N, T) gold tokens, y[:, 0] = <sos>15 N, T = y.shape16 x = y[:, 0]17 logits = []18 for t in range(1, T):19 h = self.cell(self.emb(x), h)20 z = self.out(h) # (N, vocab)21 logits.append(z)22 with torch.no_grad(): # no gradient through the choice23 pred = (torch.multinomial(F.softmax(z, -1), 1).squeeze(1)24 if sample else z.argmax(-1))25 gold = torch.rand(N, device=y.device) < eps # one coin per token26 x = torch.where(gold, y[:, t], pred)27 return torch.stack(logits, 1) # (N, T-1, vocab)2829# for i, (src, y) in enumerate(loader): # i counts mini-batches30# eps = inverse_sigmoid(i, k=5773.0) # eps = 0.5 near step 50k31# logits = dec(enc(src), y, eps)32# loss = F.cross_entropy(logits.flatten(0, 1), y[:, 1:].flatten(), ignore_index=PAD)Transformers: parallel scheduled sampling
For an RNN the coin flip costs nothing, because the decoder runs step by step anyway. A Transformer decoder trained with teacher forcing processes all positions in one parallel pass, and true scheduled sampling would force it back into a sequential loop. Parallel scheduled sampling (Duckworth, Neelakantan, Goodrich, Kaiser and Bengio, 2019) keeps the parallelism: run a teacher-forced pass, sample a token at every position from its predictions, replace each gold input by that sample with probability , and run another parallel pass on the mixed sequence, computing the loss against the gold targets. The mixed tokens are conditioned on the gold prefix rather than on the model's own earlier samples, which is the approximation that buys parallelism. The paper reports equal or better results than teacher forcing on image generation, summarisation, dialog and translation; Mihaylova and Martins (2019) proposed a similar two-pass decoder for Transformer NMT.
Related fixes attack exposure bias differently: professor forcing (Lamb et al., 2016) adds a discriminator that makes free-running hidden-state dynamics match teacher-forced ones; sequence-level training such as MIXER (Ranzato et al., 2016) optimises a test metric with REINFORCE on the model's own samples; Zhang et al. (2019) sample whole oracle sentences instead of single words.
Use when
- +Directly targets exposure bias with a two-line change to the decoder loop
- +No extra model, discriminator or reward; same cost per step for RNNs
- +Gave consistent gains on captioning, parsing and speech in the original paper
- +Robust per-token variant; a floor on ε keeps it safe
Avoid when
- −Improper objective: its optimum is not the data distribution (Huszár, 2015)
- −Breaks the time-parallel training of Transformers unless approximated (parallel scheduled sampling)
- −Schedule constants depend on training length and need tuning
- −No gradient through the sampled tokens, so it cannot learn which errors cause later errors
Check yourself
With the inverse-sigmoid schedule εᵢ = k/(k + exp(i/k)) and k = 80, at which mini-batch does the model first see its own token as often as the gold one?
Check yourself
In the two-token copy task (y₂ = y₁, both values equally likely), training ends with ε = 0. How often does the free-running model output a sequence that never occurs in the data?
Remember
- ✓Per token, feed the gold with probability , else the model's own sample; the target stays the gold .
- ✓ decays over mini-batches: linear , exponential , inverse sigmoid with midpoint .
- ✓Flip the coin per token, not per sequence; do not back-propagate through the sampled choice.
- ✓It narrows the train/inference gap (MSCOCO BLEU-4 28.8 → 30.6) but the objective is improper, so keep ε above 0.
- ✓Transformers need parallel scheduled sampling: sample from a teacher-forced pass, mix, and run a second parallel pass.