Definition
Truncated backpropagation through time (TBPTT) trains a recurrent network on a long sequence by processing it in windows. In the notation TBPTT, every time steps the network backpropagates through the most recent steps only, treating the hidden state from before the window as a constant. The forward state is carried across windows; only the gradient is cut.
Backpropagation through time (BPTT) unrolls the recurrence over the whole sequence and applies ordinary backprop to the unrolled graph (Werbos, "Backpropagation through time: what it does and how to do it", Proc. IEEE, 1990). For a sequence of length this needs all hidden states in memory and gives one parameter update per sequence. For a character-level language model trained on a 100-million-character corpus treated as one stream, or for an agent that never stops, that is impossible: memory grows without bound and learning would wait forever.
Truncation is as old as recurrent training. Elman's simple recurrent network (1990) copied the previous hidden layer into "context units" and backpropagated through one step only, effectively . Williams and Peng ("An efficient gradient-based algorithm for on-line training of recurrent network trajectories", Neural Computation, 1990) analysed truncated BPTT with a history of steps and a backward pass every steps, written BPTT in Williams and Zipser's 1995 survey. Sutskever's PhD thesis (2013) named the two knobs (forward steps between updates) and (backward steps), the notation used here. Today TBPTT with , "detach the hidden state between chunks", is the default way to train recurrent language models.
Full versus truncated gradient
Write the recurrence as with per-step losses . The full BPTT gradient sums, for every loss, the contributions of every earlier step:
- the "immediate" derivative of step , holding fixed
- the Jacobian chain that carries credit from step back to step
- the current time: an update happens whenever is a multiple of
- window length: terms with are dropped because is treated as a constant
- stride between updates; only the newest losses are in the objective, the older ones were used by earlier updates
The truncated estimate is biased: the dropped terms are exactly the ones that would teach the network to store information for longer than the window. A loss at position () reaches back only links, so with the newest loss of a window sees steps of history and the oldest sees none. Typical choices are (non-overlapping chunks, the PyTorch idiom) and (overlapping windows, so every loss gets at least steps of history).
At a glance
- Memory
- stored states instead of
- Updates
- one every steps instead of one per sequence
- Gradient
- biased: no credit assignment beyond steps
- Forward state
- not truncated: still depends on the whole past
- Compute per step
- : same as full BPTT; : each step is backpropagated (and, in autograd frameworks, re-run) times
- Typical values
- 35–200 steps for word/char LMs; AWD-LSTM uses a base of 70
The bias in practice: k2 is a hard ceiling
The echo task in the figure isolates one dependency of known length : the target at is . If , the input is not part of any computation graph that contains the loss at , so the gradient carries no information about it, and the network has no reason to keep it in its state. Training the same network on the same data while only varying makes the ceiling visible:
| schedule | lags learned (MSE < 0.1) | first lag not learned | MSE at that lag |
|---|---|---|---|
| TBPTT(4, 4) | 1–4 | 5 | 1.01 |
| TBPTT(6, 6) | 1–5 | 6 | 1.02 |
| TBPTT(3, 6) | 1–5 | 6 | 0.96 |
| TBPTT(8, 8) | 1–7 | 8 | 1.01 |
| TBPTT(2, 8) | 1–6 | 7 | 1.01 |
| TBPTT(12, 12) | 1–8 | 9 | 0.95 |
| TBPTT(1, 12) | 1–7 | 8 | 1.12 |
Two lessons. First, for small the ceiling is the window: nothing at or beyond is learned, however long you train. Second, a large window is necessary but not sufficient. With this tanh RNN still stops at because the Jacobian product in the equation shrinks with every step (see Vanilla RNN and Vanishing gradients); gated cells such as the Long short-term memory (LSTM) push that second limit out. In real language models the two limits interact: a much longer than the cell's effective memory buys little.
Worked example: what truncation removes
A scalar linear RNN with , , inputs and a single loss at the end. Then and , so . Each term with equals ; the term is because .
| k₂ | terms kept (s) | ∂h₅/∂w | ∂L/∂w | share of full gradient |
|---|---|---|---|---|
| 1 | 5 | 0.7290 | 0.4783 | 25% |
| 2 | 4, 5 | 1.4580 | 0.9566 | 50% |
| 3 | 3, 4, 5 | 2.1870 | 1.4349 | 75% |
| 4 | 2 … 5 | 2.9160 | 1.9132 | 100% |
| 5 (full) | 1 … 5 | 2.9160 | 1.9132 | 100% |
The only input arrived 4 steps before the loss. Every truncation shorter than that underestimates the gradient, here by exactly one quarter per missing step, and always in the same direction: that systematic error is the bias. Once covers the dependency the estimate is exact.
Stateful language-model training
The standard recipe (k1 = k2 = bptt)
- 1Batchify. Concatenate the corpus into one token stream and cut it into equal contiguous pieces, one per batch column: an array of shape . Column is a single long text read top to bottom.
- 2Chunk. Step through the rows in windows of
bptttokens: inputs are rows , targets the same rows shifted by one. - 3Carry and detach. Feed the final hidden (and, for an LSTM, cell) state of chunk as the initial state of chunk , after
.detach(): the values flow on, the graph is dropped, and memory stays constant. - 4Update. Backward, clip the gradient norm (see Gradient clipping), optimizer step, next chunk. Do not shuffle chunks: the carried state only makes sense if chunk continues chunk in every column.
AWD-LSTM (Merity, Keskar and Socher, "Regularizing and optimizing LSTM language models", ICLR 2018) added variable-length BPTT. With fixed windows, a token that falls at the start of a window gets almost no history in every epoch. AWD-LSTM draws the base length as bptt = 70 with probability 0.95 and 35 otherwise, then samples the actual length from a normal distribution around it with standard deviation 5, and rescales the learning rate by seq_len / bptt so short windows do not get oversized steps. It also applies DropConnect to the hidden-to-hidden weights.
1import random, torch, torch.nn as nn2import torch.nn.functional as F34def batchify(ids, B): # 1-D LongTensor of token ids -> (T, B)5 n = ids.size(0) // B6 return ids[: n * B].view(B, n).t().contiguous() # column b = one contiguous stream78def detach(state): # tensor (RNN/GRU) or (h, c) tuple (LSTM)9 return state.detach() if isinstance(state, torch.Tensor) else tuple(detach(s) for s in state)1011class LM(nn.Module):12 def __init__(self, V, E=256, H=512, layers=2):13 super().__init__()14 self.emb, self.out = nn.Embedding(V, E), nn.Linear(H, V)15 self.rnn = nn.LSTM(E, H, num_layers=layers) # sequence-first: (T, B, E)16 def forward(self, x, state=None):17 y, state = self.rnn(self.emb(x), state)18 return self.out(y), state1920def train_epoch(model, opt, data, V, bptt=70, lr=30.0, clip=0.25):21 state, i = None, 022 while i < data.size(0) - 2:23 base = bptt if random.random() < 0.95 else bptt / 2 # AWD-LSTM schedule24 seq = max(5, int(random.gauss(base, 5)))25 seq = min(seq, data.size(0) - 1 - i)26 for g in opt.param_groups:27 g["lr"] = lr * seq / bptt # rescale for the window length28 x, y = data[i : i + seq], data[i + 1 : i + 1 + seq]29 if state is not None:30 state = detach(state) # keep the values, cut the graph (k2 = seq)31 logits, state = model(x, state)32 loss = F.cross_entropy(logits.reshape(-1, V), y.reshape(-1))33 opt.zero_grad()34 loss.backward()35 nn.utils.clip_grad_norm_(model.parameters(), clip)36 opt.step()37 i += seq3839def tbptt_k1_k2(model, opt, xs, ys, V, k1, k2):40 """Overlapping windows: every k1 steps, backprop through the last k2 (k1 <= k2)."""41 state, t = None, k242 while t <= xs.size(0):43 s = t - k2 # window = steps s .. t-144 y1, mid = model(xs[s : s + k1], state) # first k1 steps of the window45 y2, _ = model(xs[s + k1 : t], mid) if k2 > k1 else (y1[:0], mid)46 out = torch.cat([y1, y2])[-k1:] # loss on the newest k1 steps only47 loss = F.cross_entropy(out.reshape(-1, V), ys[t - k1 : t].reshape(-1))48 opt.zero_grad()49 loss.backward()50 opt.step()51 state = detach(mid) # the next window starts k1 steps later52 t += k1Use when
- +Constant memory: stored states for an arbitrarily long stream
- +Frequent updates: one every steps
- +Forward state still carries the whole past, so long context is usable at test time
- +Trivial in PyTorch:
state = state.detach()between chunks
Avoid when
- −Biased gradient: dependencies of or more steps are never learned
- − costs times more backward (and re-forward) work per step
- −Requires contiguous, unshuffled batches for the carried state to be meaningful
- −Losses at the start of a window get little history unless windows overlap or vary in length
Check yourself
A character-level LSTM is trained with TBPTT, k1 = k2 = 50, carrying (detached) states between chunks. Which statement is true?
Check yourself
In the figure's echo task (target = the input L steps ago), TBPTT(6, 6) learns lags 1–5. Which change would let it learn lag 7?
Remember
- ✓TBPTT: every steps, backpropagate through the last steps; the state before the window is a constant (
detach()). - ✓Memory and an update every steps, instead of memory and one update per sequence.
- ✓The gradient is biased: terms older than the window are dropped, so dependencies of or more steps are never learned.
- ✓Only the gradient is truncated; the forward state still carries the whole past.
- ✓Stateful LM recipe: batchify into contiguous columns, chunk by
bptt, detach between chunks, clip, never shuffle chunks; AWD-LSTM varies the window around 70 and rescales the learning rate.