# Scheduled sampling

> During training, feed the decoder its own sampled token instead of the ground truth with a probability that grows over training. A per-token coin flip with a decaying ε_i narrows the train/inference gap, but the objective is not consistent.

Source: https://www.mlfoundry.in/learn/scheduled-sampling  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: advanced · 11 min read · updated 2026-10-12

**Scheduled sampling:** is a curriculum for training autoregressive decoders (Bengio, Vinyals, Jaitly and Shazeer, 2015). At every decoder step $t$ of mini-batch $i$ a coin is flipped: with probability $\varepsilon_i$ the input is the **true** previous token $y_{t-1}$ (teacher forcing), otherwise it is a token $\hat y_{t-1}$ **drawn from the model's own prediction**. $\varepsilon_i$ starts near 1 and decays toward 0 on a fixed schedule, so training slides from teacher forcing toward the conditions the model will meet at inference.

A decoder trained with **Teacher forcing** only ever sees gold prefixes. At inference it must consume its own outputs, and one early mistake puts it in a state it never visited during training, where it may keep making mistakes. Ranzato et al. (2016) later named this train/test mismatch **exposure bias**. Scheduled sampling is the simplest fix: expose the model to its own mistakes during training, but only after it is good enough for its samples to be informative.

The idea comes from imitation learning, where SEARN (Daumé, Langford and Marcu, 2009) and DAgger (Ross, Gordon and Bagnell, 2011) train a policy on the states its own actions lead to. Bengio et al. made it online and cheap: one model, one training run, no extra passes over the data. In the paper's MSCOCO captioning experiments (Table 1) a 1-layer, 512-unit LSTM went from BLEU-4 28.8 to 30.6 and CIDEr 89.5 to 92.1, and the authors report that their scheduled-sampling entry ranked first on the final leaderboard of the 2015 MSCOCO captioning challenge.

## The coin flip and the schedules

$$
x_t=\begin{cases}y_{t-1} & \text{with probability }\varepsilon_i\\[2pt] \hat y_{t-1}\sim P_\theta(\,\cdot\mid h_{t-1}) & \text{with probability }1-\varepsilon_i\end{cases}\qquad \mathcal L_i=-\sum_{t=1}^{T}\log P_\theta(y_t\mid h_t),\quad h_t=f(h_{t-1},x_t)
$$

- $i$: the mini-batch (training step) counter; the schedule is a function of training progress, not of position $t$
- $\varepsilon_i$: probability of feeding the ground truth. $\varepsilon_i=1$ is teacher forcing, $\varepsilon_i=0$ is free running
- $\hat y_{t-1}$: the model's own token: a sample from its softmax (the paper also allows the argmax)
- $y_t$: the target is always the gold token, whatever was fed as input

$$
\text{linear: }\varepsilon_i=\max(\epsilon,\;k-ci)\qquad \text{exponential: }\varepsilon_i=k^{\,i}\;(k<1)\qquad \text{inverse sigmoid: }\varepsilon_i=\frac{k}{k+\exp(i/k)}\;(k\ge1)
$$

- $\epsilon$: linear floor: the minimum amount of truth the model keeps receiving
- $k,\,c$: offset and slope (linear), base (exponential), or speed (inverse sigmoid); all set by how fast the model is expected to converge
- $i_{1/2}$: the step where $\varepsilon=0.5$: $(k-0.5)/c$, $\ln 0.5/\ln k$ and $k\ln k$ respectively

Two details from the paper matter in practice. The coin is flipped **for every token**: flipping once per sequence gave results "as poor as" always sampling, because a fully free-running sequence early in training is mostly noise and its errors compound. And the sampling decision is **not differentiated**: the gradient of the loss at later steps does not flow back through which token was picked, so each step's loss is an ordinary cross-entropy on an input that happens to be noisy.

**At a glance**

- **Paper:** Bengio, Vinyals, Jaitly, Shazeer, NeurIPS 2015
- **Fixes:** exposure bias: training on gold prefixes, testing on model prefixes
- **Knob:** schedule $\varepsilon_i$ over mini-batches; the paper used inverse sigmoid for captioning and parsing, linear ramps for speech
- **Granularity:** one coin per token (per sequence was much worse)
- **Model token:** sampled from $P_\theta(\cdot\mid h_{t-1})$, or its argmax; no gradient through the choice
- **Cost:** no extra passes for an RNN, but the decoder can no longer be run in parallel over time
- **Caveat:** improper objective (Huszár, 2015): its optimum need not be the data distribution

> Figure: Top left: the three schedules over 1000 mini-batches (inverse sigmoid with k = 80 highlighted, exponential with k = 0.995, linear with c = 0.001 and floor 0.05); the dashed tick marks where the selected schedule crosses ε = 0.5 (i = 350.6, 138.3 and 500 respectively), and the readout on the right updates live. Below: the 11 decoder inputs of one caption at the chosen iteration. Input 1 is always <s>; the other 10 each flip their own coin u and take the gold token if u < εᵢ, otherwise the stand-in model's sample (orange if it happens to be right, red if wrong). At the default i = 250, ε = 0.7785, the expected number of model inputs is 2.21 of 10 and this draw has 3, one of them wrong; P(whole sequence teacher-forced) = ε¹⁰ = 0.0818. Drag i to 500 (ε = 0.1338) and most inputs are the model's. Toggle 'coin per sequence' to see the all-or-nothing variant the paper rejected.

## How the schedules differ

The schedule decides **when** the model starts seeing its own tokens, and the three shapes spend the budget very differently. Exponential decay drops fast at the start, before the model's samples are any good. Linear decay is steady. The inverse sigmoid stays near 1 for a long warm-up, then falls steeply around $i=k\ln k$ and flattens near 0. That is why it was the paper's choice for captioning and parsing: almost pure teacher forcing while the model learns the basics, then a quick switch.

εᵢ for the figure's default schedules (computed in Node from the figure's lib)

| schedule | i = 0 | i = 250 | i = 500 | i = 750 | i = 1000 | ε = 0.5 at |
| --- | --- | --- | --- | --- | --- | --- |
| linear, $c=0.001$, floor 0.05 | 1.0000 | 0.7500 | 0.5000 | 0.2500 | 0.0500 | 500.0 |
| exponential, $k=0.995$ | 1.0000 | 0.2856 | 0.0816 | 0.0233 | 0.0067 | 138.3 |
| inverse sigmoid, $k=80$ | 0.9877 | 0.7785 | 0.1338 | 0.0067 | 0.0003 | $80\ln 80=350.6$ |

The constants have to be scaled to the real length of training. For an inverse sigmoid to cross 0.5 at step 50 000 you need $k\ln k=50\,000$, i.e. $k\approx5\,773$. Averaged over the first 1000 steps, the default inverse sigmoid feeds model tokens at 64.8% of the inputs (the 'model share so far' readout), the exponential at 80.2% and the linear at 49.9%.

## Why it helps, and why it is not a proper objective

When the input is a model sample, the target is still the gold token. The model is therefore trained to map **slightly wrong prefixes back onto the correct continuation**, which is exactly the recovery behaviour it needs at inference. The paper's speech experiment shows the trade-off cleanly (Table 3, TIMIT frame error rate): the teacher-forced baseline has the best next-step error with gold history (15.0%) but the worst decoding error (46.0%), while linear schedules from 0.25 or 0.5 down to 0 reach 34.5% and 35.0% when decoding.

Huszár (2015) showed the price: the scheduled-sampling objective is **improper**, so even with infinite data and capacity its minimiser is generally not the data distribution, and the learning algorithm is inconsistent. The reason is visible in the loss. When the input $\hat y_{t-1}$ was sampled independently of the gold prefix, the target $y_t$ carries information about the gold token that the input does not, and the best the model can do is predict a blend of the true conditional and the marginal.

**Worked example: a two-token copy task (checked by Monte Carlo in Node)**

1. Data: $y_1\in\{A,B\}$ with probability $\tfrac12$ each, and $y_2=y_1$. Only $AA$ and $BB$ ever occur. Step 1 is always fed <s>, so the model learns $q(y_1)=\tfrac12$ exactly.
2. Step 2 with input $a$: with probability $\varepsilon$ the input is the gold $y_1$, so the target is $a$. With probability $1-\varepsilon$ it is the model's sample, independent of the gold $y_1$, so the target equals $a$ only half the time.
3. The cross-entropy minimiser is the conditional frequency: $q(y_2=a\mid a)=\dfrac{\varepsilon\cdot\tfrac12+(1-\varepsilon)\cdot\tfrac12\cdot\tfrac12}{\tfrac12}=\dfrac{1+\varepsilon}{2}$.
4. At the end of training: $\varepsilon=1$ gives 1 (correct); $\varepsilon=0.5$ gives 0.75; $\varepsilon=0$ gives 0.5. A simulation with $10^6$ sequences gives 0.7502 and 0.4999.
5. Free-running, the $\varepsilon=0$ model generates the impossible sequences $AB$ or $BA$ with probability $(1-\varepsilon)/2=0.5$. Driving $\varepsilon$ all the way to 0 has taught it to ignore its own history.

> **Recovery versus consistency:** Both effects in the example come from the same mechanism. A gold target paired with a noisy input teaches recovery when the noise is a **small** perturbation of the gold prefix (a near-miss word, a slightly wrong frame label). When the model's sample is genuinely different, the same pairing teaches the model to discount its input. In practice this is why the schedule usually stops short of 0, and why scheduled sampling works best when the model is already accurate.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

def inverse_sigmoid(i, k):          # eps_i = k / (k + exp(i/k)); 0.5 at i = k ln k
    return k / (k + math.exp(min(i / k, 50.0)))

class Decoder(nn.Module):
    def __init__(self, vocab, emb=256, hid=512):
        super().__init__()
        self.emb = nn.Embedding(vocab, emb)
        self.cell = nn.GRUCell(emb, hid)
        self.out = nn.Linear(hid, vocab)

    def forward(self, h, y, eps, sample=True):
        # h: (N, hid) initial state from the encoder; y: (N, T) gold tokens, y[:, 0] = <sos>
        N, T = y.shape
        x = y[:, 0]
        logits = []
        for t in range(1, T):
            h = self.cell(self.emb(x), h)
            z = self.out(h)                                   # (N, vocab)
            logits.append(z)
            with torch.no_grad():                             # no gradient through the choice
                pred = (torch.multinomial(F.softmax(z, -1), 1).squeeze(1)
                        if sample else z.argmax(-1))
                gold = torch.rand(N, device=y.device) < eps   # one coin per token
                x = torch.where(gold, y[:, t], pred)
        return torch.stack(logits, 1)                         # (N, T-1, vocab)

# for i, (src, y) in enumerate(loader):                       # i counts mini-batches
#     eps = inverse_sigmoid(i, k=5773.0)                       # eps = 0.5 near step 50k
#     logits = dec(enc(src), y, eps)
#     loss = F.cross_entropy(logits.flatten(0, 1), y[:, 1:].flatten(), ignore_index=PAD)
```

A GRU decoder with per-token scheduled sampling and an inverse-sigmoid schedule

## Transformers: parallel scheduled sampling

For an RNN the coin flip costs nothing, because the decoder runs step by step anyway. A Transformer decoder trained with teacher forcing processes all positions in one parallel pass, and true scheduled sampling would force it back into a sequential loop. **Parallel scheduled sampling** (Duckworth, Neelakantan, Goodrich, Kaiser and Bengio, 2019) keeps the parallelism: run a teacher-forced pass, sample a token at every position from its predictions, replace each gold input by that sample with probability $1-\varepsilon_i$, and run another parallel pass on the mixed sequence, computing the loss against the gold targets. The mixed tokens are conditioned on the gold prefix rather than on the model's own earlier samples, which is the approximation that buys parallelism. The paper reports equal or better results than teacher forcing on image generation, summarisation, dialog and translation; Mihaylova and Martins (2019) proposed a similar two-pass decoder for Transformer NMT.

Related fixes attack exposure bias differently: **professor forcing** (Lamb et al., 2016) adds a discriminator that makes free-running hidden-state dynamics match teacher-forced ones; sequence-level training such as MIXER (Ranzato et al., 2016) optimises a test metric with REINFORCE on the model's own samples; Zhang et al. (2019) sample whole oracle sentences instead of single words.

> **Don't decay to zero, and tune k to your step count:** Copying the paper's constants into a run with a different number of steps either never leaves teacher forcing or abandons it before the model is useful. Pick the step where $\varepsilon$ should cross 0.5 and solve for $k$. Keep a floor (the linear schedule's $\epsilon$), since the copy example shows that $\varepsilon\to0$ actively teaches the model to ignore its history. Always report the free-running metric (BLEU, WER, CIDEr), not the teacher-forced loss, which scheduled sampling makes **worse** by design.

> **Sample or argmax:** The paper allows either. Sampling exposes the model to the variety of errors it will actually make if you decode by sampling; the argmax matches greedy or beam decoding (see **Beam search**) and has lower variance. Either way, wrap the choice in `torch.no_grad()` or call `.detach()`: the paper did not back-propagate through the sampling decision.

**Strengths**

- Directly targets exposure bias with a two-line change to the decoder loop
- No extra model, discriminator or reward; same cost per step for RNNs
- Gave consistent gains on captioning, parsing and speech in the original paper
- Robust per-token variant; a floor on ε keeps it safe

**Limitations**

- Improper objective: its optimum is not the data distribution (Huszár, 2015)
- Breaks the time-parallel training of Transformers unless approximated (parallel scheduled sampling)
- Schedule constants depend on training length and need tuning
- No gradient through the sampled tokens, so it cannot learn which errors cause later errors

**Question:** With the inverse-sigmoid schedule εᵢ = k/(k + exp(i/k)) and k = 80, at which mini-batch does the model first see its own token as often as the gold one?

- A. i = 80
- B. i ≈ 351
- C. i = 500
- D. i ≈ 6400

**Answer:** B. i ≈ 351 — $\varepsilon_i=\tfrac12$ when $e^{i/k}=k$, i.e. $i=k\ln k=80\ln 80=350.6$. Before that the model is mostly teacher-forced; after it, mostly fed its own samples.

**Question:** In the two-token copy task (y₂ = y₁, both values equally likely), training ends with ε = 0. How often does the free-running model output a sequence that never occurs in the data?

- A. Never
- B. 12.5% of the time
- C. 25% of the time
- D. 50% of the time

**Answer:** D. 50% of the time — The optimum is $q(y_2=a\mid a)=(1+\varepsilon)/2$. With $\varepsilon=0$ the sampled input is independent of the gold target, so the model learns $q(y_2\mid a)=\tfrac12$ and produces $AB$ or $BA$ half the time. This is Huszár's inconsistency in miniature.

**Key points**

- Per token, feed the gold $y_{t-1}$ with probability $\varepsilon_i$, else the model's own sample; the target stays the gold $y_t$.
- $\varepsilon_i$ decays over mini-batches: linear $\max(\epsilon,k-ci)$, exponential $k^i$, inverse sigmoid $k/(k+e^{i/k})$ with midpoint $k\ln k$.
- Flip the coin per token, not per sequence; do not back-propagate through the sampled choice.
- It narrows the train/inference gap (MSCOCO BLEU-4 28.8 → 30.6) but the objective is improper, so keep ε above 0.
- Transformers need parallel scheduled sampling: sample from a teacher-forced pass, mix, and run a second parallel pass.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
