Truncated backpropagation through time (TBPTT)

Run the RNN forward over an endless stream but backpropagate only through the last k2 steps, updating every k1 steps. Memory becomes constant and updates frequent, at the price of a biased gradient: a dependency longer than k2 steps is never learned.

11 min read1204 wordscore0 viewsupdated 2026-10-12

Definition

Truncated backpropagation through time (TBPTT) trains a recurrent network on a long sequence by processing it in windows. In the notation TBPTT(k1,k2)(k_1,k_2), every k1k_1 time steps the network backpropagates through the most recent k2≥k1k_2\ge k_1 steps only, treating the hidden state from before the window as a constant. The forward state is carried across windows; only the gradient is cut.

Backpropagation through time (BPTT) unrolls the recurrence over the whole sequence and applies ordinary backprop to the unrolled graph (Werbos, "Backpropagation through time: what it does and how to do it", Proc. IEEE, 1990). For a sequence of length TT this needs all TT hidden states in memory and gives one parameter update per sequence. For a character-level language model trained on a 100-million-character corpus treated as one stream, or for an agent that never stops, that is impossible: memory grows without bound and learning would wait forever.

Truncation is as old as recurrent training. Elman's simple recurrent network (1990) copied the previous hidden layer into "context units" and backpropagated through one step only, effectively k2=1k_2=1. Williams and Peng ("An efficient gradient-based algorithm for on-line training of recurrent network trajectories", Neural Computation, 1990) analysed truncated BPTT with a history of hh steps and a backward pass every h′h' steps, written BPTT(h;h′)(h;h') in Williams and Zipser's 1995 survey. Sutskever's PhD thesis (2013) named the two knobs k1k_1 (forward steps between updates) and k2k_2 (backward steps), the notation used here. Today TBPTT with k1=k2k_1=k_2, "detach the hidden state between chunks", is the default way to train recurrent language models.

Full versus truncated gradient

Write the recurrence as ht=Fθ(ht−1,xt)h_t=F_\theta(h_{t-1},x_t) with per-step losses Lt\mathcal L_t. The full BPTT gradient sums, for every loss, the contributions of every earlier step:

∂L∂θ=∑t∑s≤t∂Lt∂ht(∏j=s+1t∂hj∂hj−1)∂+hs∂θ⟶∂L∂θ^=∑t∈last k1 ∑s=τ−k2+1t∂Lt∂ht(∏j=s+1t∂hj∂hj−1)∂+hs∂θ\frac{\partial\mathcal L}{\partial\theta}=\sum_{t}\sum_{s\le t}\frac{\partial\mathcal L_t}{\partial h_t}\Bigg(\prod_{j=s+1}^{t}\frac{\partial h_j}{\partial h_{j-1}}\Bigg)\frac{\partial^{+}h_s}{\partial\theta}\qquad\longrightarrow\qquad \widehat{\frac{\partial\mathcal L}{\partial\theta}}=\sum_{t\in\text{last }k_1}\ \sum_{s=\tau-k_2+1}^{t}\frac{\partial\mathcal L_t}{\partial h_t}\Bigg(\prod_{j=s+1}^{t}\frac{\partial h_j}{\partial h_{j-1}}\Bigg)\frac{\partial^{+}h_s}{\partial\theta}
∂+hs/∂θ\partial^{+}h_s/\partial\theta
the "immediate" derivative of step ss, holding hs−1h_{s-1} fixed
∏∂hj/∂hj−1\prod\partial h_j/\partial h_{j-1}
the Jacobian chain that carries credit from step tt back to step ss
τ\tau
the current time: an update happens whenever τ\tau is a multiple of k1k_1
k2k_2
window length: terms with s≤τ−k2s\le\tau-k_2 are dropped because hτ−k2h_{\tau-k_2} is treated as a constant
k1k_1
stride between updates; only the newest k1k_1 losses are in the objective, the older ones were used by earlier updates

The truncated estimate is biased: the dropped terms are exactly the ones that would teach the network to store information for longer than the window. A loss at position t=τ−jt=\tau-j (j=0,…,k1−1j=0,\dots,k_1-1) reaches back only k2−1−jk_2-1-j links, so with k1=k2=kk_1=k_2=k the newest loss of a window sees k−1k-1 steps of history and the oldest sees none. Typical choices are k1=k2k_1=k_2 (non-overlapping chunks, the PyTorch idiom) and k1<k2k_1<k_2 (overlapping windows, so every loss gets at least k2−k1k_2-k_1 steps of history).

At a glance

Memory
O(k2)O(k_2) stored states instead of O(T)O(T)
Updates
one every k1k_1 steps instead of one per sequence
Gradient
biased: no credit assignment beyond k2k_2 steps
Forward state
not truncated: hτh_\tau still depends on the whole past
Compute per step
k1=k2k_1=k_2: same as full BPTT; k1<k2k_1<k_2: each step is backpropagated (and, in autograd frameworks, re-run) k2/k1k_2/k_1 times
Typical values
35–200 steps for word/char LMs; AWD-LSTM uses a base of 70
stream xₜ, target yₜ = xₜ₋4: TBPTT(6, 6) runs every 6 steps and backprops 6 steps■ k2 = 6 steps in the graph ■ k1 = 6 steps with loss (click one) older updates fadeddetach()01234567891011121314151617181920212223loss at t = 23 needs x at t = 19: inside the graph (4 links back)loss positions whose lag-4 input is reachable: 2/6 = 33%cost: 6 states stored · 1.00 fwd + 1.00 bwd steps per step · 166 updates/1000 stepstraining MSE, 12-unit tanh RNN, 8 streams, Adamfinal 0.000 · sign acc 100.0% · learnedpredict 0 → MSE 100.51600 updatesfinal MSE vs lag L at TBPTT(6, 6)123456789101112press “sweep lags” (12 trainings)
red tick = k2
fig.Top: the stream, the windows of the last three TBPTT(k1, k2) updates, and the gradient path (violet arcs) of the selected loss back to the detach() line. Bottom: a 12-unit tanh RNN trained live with Adam (600 updates, 8 streams) to output the input from L steps ago. At the default TBPTT(6, 6) with lag 4, only 2 of the 6 losses per window can reach x(t−4), yet that is enough: MSE drops from 0.95 to 0.000. Sweep the lags: lags 1–5 are learned and every lag ≥ 6 = k2 stays at MSE ≈ 1, the score of always predicting 0. With k2 = 12 the limit is lag 8, not 11: there the vanishing gradient of the tanh cell, not the truncation, is what binds.

The bias in practice: k2 is a hard ceiling

The echo task in the figure isolates one dependency of known length LL: the target at tt is xt−Lx_{t-L}. If L≥k2L\ge k_2, the input xt−Lx_{t-L} is not part of any computation graph that contains the loss at tt, so the gradient carries no information about it, and the network has no reason to keep it in its state. Training the same network on the same data while only varying LL makes the ceiling visible:

schedulelags learned (MSE < 0.1)first lag not learnedMSE at that lag
TBPTT(4, 4)1–451.01
TBPTT(6, 6)1–561.02
TBPTT(3, 6)1–560.96
TBPTT(8, 8)1–781.01
TBPTT(2, 8)1–671.01
TBPTT(12, 12)1–890.95
TBPTT(1, 12)1–781.12
Final training MSE after 600 updates (12-unit tanh RNN, Adam, lr 0.02, seed 7; computed by the figure's library). 1.0 = chance.

Two lessons. First, for small k2k_2 the ceiling is the window: nothing at or beyond L=k2L=k_2 is learned, however long you train. Second, a large window is necessary but not sufficient. With k2=12k_2=12 this tanh RNN still stops at L=8L=8 because the Jacobian product in the equation shrinks with every step (see Vanilla RNN and Vanishing gradients); gated cells such as the Long short-term memory (LSTM) push that second limit out. In real language models the two limits interact: a k2k_2 much longer than the cell's effective memory buys little.

Worked example: what truncation removes

A scalar linear RNN ht=w ht−1+xth_t=w\,h_{t-1}+x_t with h0=0h_0=0, w=0.9w=0.9, inputs x=(1,0,0,0,0)x=(1,0,0,0,0) and a single loss L=12h52\mathcal L=\tfrac12h_5^2 at the end. Then h=(1, 0.9, 0.81, 0.729, 0.6561)h=(1,\,0.9,\,0.81,\,0.729,\,0.6561) and ∂ht/∂w=ht−1+w ∂ht−1/∂w\partial h_t/\partial w=h_{t-1}+w\,\partial h_{t-1}/\partial w, so ∂h5/∂w=∑s=15w5−shs−1\partial h_5/\partial w=\sum_{s=1}^{5}w^{5-s}h_{s-1}. Each term with s≥2s\ge2 equals 0.95−s⋅0.9s−2=0.7290.9^{5-s}\cdot0.9^{s-2}=0.729; the s=1s=1 term is 00 because h0=0h_0=0.

k₂terms kept (s)∂h₅/∂w∂L/∂wshare of full gradient
150.72900.478325%
24, 51.45800.956650%
33, 4, 52.18701.434975%
42 … 52.91601.9132100%
5 (full)1 … 52.91601.9132100%
Gradient ∂L/∂w = h₅ · ∂h₅/∂w when backpropagating only k₂ steps (checked in Node; full gradient confirmed by finite differences)

The only input arrived 4 steps before the loss. Every truncation shorter than that underestimates the gradient, here by exactly one quarter per missing step, and always in the same direction: that systematic error is the bias. Once k2k_2 covers the dependency the estimate is exact.

Stateful language-model training

The standard recipe (k1 = k2 = bptt)

  1. 1Batchify. Concatenate the corpus into one token stream and cut it into BB equal contiguous pieces, one per batch column: an array of shape (T/B, B)(T/B,\,B). Column bb is a single long text read top to bottom.
  2. 2Chunk. Step through the rows in windows of bptt tokens: inputs are rows i…i+bptt−1i\dots i+\text{bptt}-1, targets the same rows shifted by one.
  3. 3Carry and detach. Feed the final hidden (and, for an LSTM, cell) state of chunk nn as the initial state of chunk n+1n+1, after .detach(): the values flow on, the graph is dropped, and memory stays constant.
  4. 4Update. Backward, clip the gradient norm (see Gradient clipping), optimizer step, next chunk. Do not shuffle chunks: the carried state only makes sense if chunk n+1n+1 continues chunk nn in every column.

AWD-LSTM (Merity, Keskar and Socher, "Regularizing and optimizing LSTM language models", ICLR 2018) added variable-length BPTT. With fixed windows, a token that falls at the start of a window gets almost no history in every epoch. AWD-LSTM draws the base length as bptt = 70 with probability 0.95 and 35 otherwise, then samples the actual length from a normal distribution around it with standard deviation 5, and rescales the learning rate by seq_len / bptt so short windows do not get oversized steps. It also applies DropConnect to the hidden-to-hidden weights.

python
1import random, torch, torch.nn as nn
2import torch.nn.functional as F
3
4def batchify(ids, B): # 1-D LongTensor of token ids -> (T, B)
5 n = ids.size(0) // B
6 return ids[: n * B].view(B, n).t().contiguous() # column b = one contiguous stream
7
8def detach(state): # tensor (RNN/GRU) or (h, c) tuple (LSTM)
9 return state.detach() if isinstance(state, torch.Tensor) else tuple(detach(s) for s in state)
10
11class LM(nn.Module):
12 def __init__(self, V, E=256, H=512, layers=2):
13 super().__init__()
14 self.emb, self.out = nn.Embedding(V, E), nn.Linear(H, V)
15 self.rnn = nn.LSTM(E, H, num_layers=layers) # sequence-first: (T, B, E)
16 def forward(self, x, state=None):
17 y, state = self.rnn(self.emb(x), state)
18 return self.out(y), state
19
20def train_epoch(model, opt, data, V, bptt=70, lr=30.0, clip=0.25):
21 state, i = None, 0
22 while i < data.size(0) - 2:
23 base = bptt if random.random() < 0.95 else bptt / 2 # AWD-LSTM schedule
24 seq = max(5, int(random.gauss(base, 5)))
25 seq = min(seq, data.size(0) - 1 - i)
26 for g in opt.param_groups:
27 g["lr"] = lr * seq / bptt # rescale for the window length
28 x, y = data[i : i + seq], data[i + 1 : i + 1 + seq]
29 if state is not None:
30 state = detach(state) # keep the values, cut the graph (k2 = seq)
31 logits, state = model(x, state)
32 loss = F.cross_entropy(logits.reshape(-1, V), y.reshape(-1))
33 opt.zero_grad()
34 loss.backward()
35 nn.utils.clip_grad_norm_(model.parameters(), clip)
36 opt.step()
37 i += seq
38
39def tbptt_k1_k2(model, opt, xs, ys, V, k1, k2):
40 """Overlapping windows: every k1 steps, backprop through the last k2 (k1 <= k2)."""
41 state, t = None, k2
42 while t <= xs.size(0):
43 s = t - k2 # window = steps s .. t-1
44 y1, mid = model(xs[s : s + k1], state) # first k1 steps of the window
45 y2, _ = model(xs[s + k1 : t], mid) if k2 > k1 else (y1[:0], mid)
46 out = torch.cat([y1, y2])[-k1:] # loss on the newest k1 steps only
47 loss = F.cross_entropy(out.reshape(-1, V), ys[t - k1 : t].reshape(-1))
48 opt.zero_grad()
49 loss.backward()
50 opt.step()
51 state = detach(mid) # the next window starts k1 steps later
52 t += k1
TBPTT(k, k) with detach(), AWD-style variable windows, and a general TBPTT(k1, k2)

Use when

  • +Constant memory: O(k2)O(k_2) stored states for an arbitrarily long stream
  • +Frequent updates: one every k1k_1 steps
  • +Forward state still carries the whole past, so long context is usable at test time
  • +Trivial in PyTorch: state = state.detach() between chunks

Avoid when

  • −Biased gradient: dependencies of k2k_2 or more steps are never learned
  • −k1<k2k_1<k_2 costs k2/k1k_2/k_1 times more backward (and re-forward) work per step
  • −Requires contiguous, unshuffled batches for the carried state to be meaningful
  • −Losses at the start of a window get little history unless windows overlap or vary in length

Check yourself

A character-level LSTM is trained with TBPTT, k1 = k2 = 50, carrying (detached) states between chunks. Which statement is true?

Check yourself

In the figure's echo task (target = the input L steps ago), TBPTT(6, 6) learns lags 1–5. Which change would let it learn lag 7?

Remember

  • ✓TBPTT(k1,k2)(k_1,k_2): every k1k_1 steps, backpropagate through the last k2k_2 steps; the state before the window is a constant (detach()).
  • ✓Memory O(k2)O(k_2) and an update every k1k_1 steps, instead of O(T)O(T) memory and one update per sequence.
  • ✓The gradient is biased: terms older than the window are dropped, so dependencies of k2k_2 or more steps are never learned.
  • ✓Only the gradient is truncated; the forward state still carries the whole past.
  • ✓Stateful LM recipe: batchify into contiguous columns, chunk by bptt, detach between chunks, clip, never shuffle chunks; AWD-LSTM varies the window around 70 and rescales the learning rate.
← back to Deep Learning roadmap