# Truncated backpropagation through time (TBPTT)

> Run the RNN forward over an endless stream but backpropagate only through the last k2 steps, updating every k1 steps. Memory becomes constant and updates frequent, at the price of a biased gradient: a dependency longer than k2 steps is never learned.

Source: https://www.mlfoundry.in/learn/truncated-backpropagation-through-time-tbptt  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 11 min read · updated 2026-10-12

**Truncated backpropagation through time:** (TBPTT) trains a recurrent network on a long sequence by processing it in windows. In the notation TBPTT$(k_1,k_2)$, every $k_1$ time steps the network backpropagates through the most recent $k_2\ge k_1$ steps only, treating the hidden state from before the window as a constant. The forward state is carried across windows; only the **gradient** is cut.

Backpropagation through time (BPTT) unrolls the recurrence over the whole sequence and applies ordinary backprop to the unrolled graph (Werbos, "Backpropagation through time: what it does and how to do it", *Proc. IEEE*, 1990). For a sequence of length $T$ this needs all $T$ hidden states in memory and gives **one** parameter update per sequence. For a character-level language model trained on a 100-million-character corpus treated as one stream, or for an agent that never stops, that is impossible: memory grows without bound and learning would wait forever.

Truncation is as old as recurrent training. Elman's simple recurrent network (1990) copied the previous hidden layer into "context units" and backpropagated through **one** step only, effectively $k_2=1$. Williams and Peng ("An efficient gradient-based algorithm for on-line training of recurrent network trajectories", *Neural Computation*, 1990) analysed truncated BPTT with a history of $h$ steps and a backward pass every $h'$ steps, written BPTT$(h;h')$ in Williams and Zipser's 1995 survey. Sutskever's PhD thesis (2013) named the two knobs $k_1$ (forward steps between updates) and $k_2$ (backward steps), the notation used here. Today TBPTT with $k_1=k_2$, "detach the hidden state between chunks", is the default way to train recurrent language models.

## Full versus truncated gradient

Write the recurrence as $h_t=F_\theta(h_{t-1},x_t)$ with per-step losses $\mathcal L_t$. The full BPTT gradient sums, for every loss, the contributions of every earlier step:

$$
\frac{\partial\mathcal L}{\partial\theta}=\sum_{t}\sum_{s\le t}\frac{\partial\mathcal L_t}{\partial h_t}\Bigg(\prod_{j=s+1}^{t}\frac{\partial h_j}{\partial h_{j-1}}\Bigg)\frac{\partial^{+}h_s}{\partial\theta}\qquad\longrightarrow\qquad \widehat{\frac{\partial\mathcal L}{\partial\theta}}=\sum_{t\in\text{last }k_1}\ \sum_{s=\tau-k_2+1}^{t}\frac{\partial\mathcal L_t}{\partial h_t}\Bigg(\prod_{j=s+1}^{t}\frac{\partial h_j}{\partial h_{j-1}}\Bigg)\frac{\partial^{+}h_s}{\partial\theta}
$$

- $\partial^{+}h_s/\partial\theta$: the "immediate" derivative of step $s$, holding $h_{s-1}$ fixed
- $\prod\partial h_j/\partial h_{j-1}$: the Jacobian chain that carries credit from step $t$ back to step $s$
- $\tau$: the current time: an update happens whenever $\tau$ is a multiple of $k_1$
- $k_2$: window length: terms with $s\le\tau-k_2$ are dropped because $h_{\tau-k_2}$ is treated as a constant
- $k_1$: stride between updates; only the newest $k_1$ losses are in the objective, the older ones were used by earlier updates

The truncated estimate is **biased**: the dropped terms are exactly the ones that would teach the network to store information for longer than the window. A loss at position $t=\tau-j$ ($j=0,\dots,k_1-1$) reaches back only $k_2-1-j$ links, so with $k_1=k_2=k$ the newest loss of a window sees $k-1$ steps of history and the oldest sees none. Typical choices are $k_1=k_2$ (non-overlapping chunks, the PyTorch idiom) and $k_1<k_2$ (overlapping windows, so every loss gets at least $k_2-k_1$ steps of history).

**At a glance**

- **Memory:** $O(k_2)$ stored states instead of $O(T)$
- **Updates:** one every $k_1$ steps instead of one per sequence
- **Gradient:** biased: no credit assignment beyond $k_2$ steps
- **Forward state:** not truncated: $h_\tau$ still depends on the whole past
- **Compute per step:** $k_1=k_2$: same as full BPTT; $k_1<k_2$: each step is backpropagated (and, in autograd frameworks, re-run) $k_2/k_1$ times
- **Typical values:** 35–200 steps for word/char LMs; AWD-LSTM uses a base of 70

> Figure: Top: the stream, the windows of the last three TBPTT(k1, k2) updates, and the gradient path (violet arcs) of the selected loss back to the detach() line. Bottom: a 12-unit tanh RNN trained live with Adam (600 updates, 8 streams) to output the input from L steps ago. At the default TBPTT(6, 6) with lag 4, only 2 of the 6 losses per window can reach x(t−4), yet that is enough: MSE drops from 0.95 to 0.000. Sweep the lags: lags 1–5 are learned and every lag ≥ 6 = k2 stays at MSE ≈ 1, the score of always predicting 0. With k2 = 12 the limit is lag 8, not 11: there the vanishing gradient of the tanh cell, not the truncation, is what binds.

## The bias in practice: k2 is a hard ceiling

The echo task in the figure isolates one dependency of known length $L$: the target at $t$ is $x_{t-L}$. If $L\ge k_2$, the input $x_{t-L}$ is not part of any computation graph that contains the loss at $t$, so the gradient carries no information about it, and the network has no reason to keep it in its state. Training the same network on the same data while only varying $L$ makes the ceiling visible:

Final training MSE after 600 updates (12-unit tanh RNN, Adam, lr 0.02, seed 7; computed by the figure's library). 1.0 = chance.

| schedule | lags learned (MSE < 0.1) | first lag not learned | MSE at that lag |
| --- | --- | --- | --- |
| TBPTT(4, 4) | 1–4 | 5 | 1.01 |
| TBPTT(6, 6) | 1–5 | 6 | 1.02 |
| TBPTT(3, 6) | 1–5 | 6 | 0.96 |
| TBPTT(8, 8) | 1–7 | 8 | 1.01 |
| TBPTT(2, 8) | 1–6 | 7 | 1.01 |
| TBPTT(12, 12) | 1–8 | 9 | 0.95 |
| TBPTT(1, 12) | 1–7 | 8 | 1.12 |

Two lessons. First, for small $k_2$ the ceiling is the window: nothing at or beyond $L=k_2$ is learned, however long you train. Second, a large window is necessary but not sufficient. With $k_2=12$ this tanh RNN still stops at $L=8$ because the Jacobian product in the equation shrinks with every step (see **Vanilla RNN** and **Vanishing gradients**); gated cells such as the **Long short-term memory (LSTM)** push that second limit out. In real language models the two limits interact: a $k_2$ much longer than the cell's effective memory buys little.

> **The forward pass is not truncated:** Only the gradient is cut. The state handed to the next window is the real $h_{\tau}$, which depends on the entire past, so a TBPTT-trained LSTM can and does *use* information older than $k_2$ at test time, provided that something it learned for shorter dependencies happens to preserve it. What it cannot do is *learn* to preserve information specifically because it will be useful more than $k_2$ steps later.

## Worked example: what truncation removes

A scalar linear RNN $h_t=w\,h_{t-1}+x_t$ with $h_0=0$, $w=0.9$, inputs $x=(1,0,0,0,0)$ and a single loss $\mathcal L=\tfrac12h_5^2$ at the end. Then $h=(1,\,0.9,\,0.81,\,0.729,\,0.6561)$ and $\partial h_t/\partial w=h_{t-1}+w\,\partial h_{t-1}/\partial w$, so $\partial h_5/\partial w=\sum_{s=1}^{5}w^{5-s}h_{s-1}$. Each term with $s\ge2$ equals $0.9^{5-s}\cdot0.9^{s-2}=0.729$; the $s=1$ term is $0$ because $h_0=0$.

Gradient ∂L/∂w = h₅ · ∂h₅/∂w when backpropagating only k₂ steps (checked in Node; full gradient confirmed by finite differences)

| k₂ | terms kept (s) | ∂h₅/∂w | ∂L/∂w | share of full gradient |
| --- | --- | --- | --- | --- |
| 1 | 5 | 0.7290 | 0.4783 | 25% |
| 2 | 4, 5 | 1.4580 | 0.9566 | 50% |
| 3 | 3, 4, 5 | 2.1870 | 1.4349 | 75% |
| 4 | 2 … 5 | 2.9160 | 1.9132 | 100% |
| 5 (full) | 1 … 5 | 2.9160 | 1.9132 | 100% |

The only input arrived 4 steps before the loss. Every truncation shorter than that underestimates the gradient, here by exactly one quarter per missing step, and always in the same direction: that systematic error is the bias. Once $k_2$ covers the dependency the estimate is exact.

## Stateful language-model training

**The standard recipe (k1 = k2 = bptt)**

1. **Batchify.** Concatenate the corpus into one token stream and cut it into $B$ equal contiguous pieces, one per batch column: an array of shape $(T/B,\,B)$. Column $b$ is a single long text read top to bottom.
2. **Chunk.** Step through the rows in windows of `bptt` tokens: inputs are rows $i\dots i+\text{bptt}-1$, targets the same rows shifted by one.
3. **Carry and detach.** Feed the final hidden (and, for an LSTM, cell) state of chunk $n$ as the initial state of chunk $n+1$, after `.detach()`: the values flow on, the graph is dropped, and memory stays constant.
4. **Update.** Backward, clip the gradient norm (see **Gradient clipping**), optimizer step, next chunk. Do not shuffle chunks: the carried state only makes sense if chunk $n+1$ continues chunk $n$ in every column.

AWD-LSTM (Merity, Keskar and Socher, "Regularizing and optimizing LSTM language models", ICLR 2018) added **variable-length BPTT**. With fixed windows, a token that falls at the start of a window gets almost no history in every epoch. AWD-LSTM draws the base length as `bptt` = 70 with probability 0.95 and 35 otherwise, then samples the actual length from a normal distribution around it with standard deviation 5, and rescales the learning rate by `seq_len / bptt` so short windows do not get oversized steps. It also applies **DropConnect** to the hidden-to-hidden weights.

```python
import random, torch, torch.nn as nn
import torch.nn.functional as F

def batchify(ids, B):                       # 1-D LongTensor of token ids -> (T, B)
    n = ids.size(0) // B
    return ids[: n * B].view(B, n).t().contiguous()   # column b = one contiguous stream

def detach(state):                          # tensor (RNN/GRU) or (h, c) tuple (LSTM)
    return state.detach() if isinstance(state, torch.Tensor) else tuple(detach(s) for s in state)

class LM(nn.Module):
    def __init__(self, V, E=256, H=512, layers=2):
        super().__init__()
        self.emb, self.out = nn.Embedding(V, E), nn.Linear(H, V)
        self.rnn = nn.LSTM(E, H, num_layers=layers)   # sequence-first: (T, B, E)
    def forward(self, x, state=None):
        y, state = self.rnn(self.emb(x), state)
        return self.out(y), state

def train_epoch(model, opt, data, V, bptt=70, lr=30.0, clip=0.25):
    state, i = None, 0
    while i < data.size(0) - 2:
        base = bptt if random.random() < 0.95 else bptt / 2       # AWD-LSTM schedule
        seq = max(5, int(random.gauss(base, 5)))
        seq = min(seq, data.size(0) - 1 - i)
        for g in opt.param_groups:
            g["lr"] = lr * seq / bptt                             # rescale for the window length
        x, y = data[i : i + seq], data[i + 1 : i + 1 + seq]
        if state is not None:
            state = detach(state)            # keep the values, cut the graph (k2 = seq)
        logits, state = model(x, state)
        loss = F.cross_entropy(logits.reshape(-1, V), y.reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), clip)
        opt.step()
        i += seq

def tbptt_k1_k2(model, opt, xs, ys, V, k1, k2):
    """Overlapping windows: every k1 steps, backprop through the last k2 (k1 <= k2)."""
    state, t = None, k2
    while t <= xs.size(0):
        s = t - k2                                         # window = steps s .. t-1
        y1, mid = model(xs[s : s + k1], state)             # first k1 steps of the window
        y2, _ = model(xs[s + k1 : t], mid) if k2 > k1 else (y1[:0], mid)
        out = torch.cat([y1, y2])[-k1:]                    # loss on the newest k1 steps only
        loss = F.cross_entropy(out.reshape(-1, V), ys[t - k1 : t].reshape(-1))
        opt.zero_grad()
        loss.backward()
        opt.step()
        state = detach(mid)                                # the next window starts k1 steps later
        t += k1
```

TBPTT(k, k) with detach(), AWD-style variable windows, and a general TBPTT(k1, k2)

> **Forgetting detach(), or detaching too much:** If the carried state is not detached, the second `loss.backward()` tries to go back into the previous chunk's graph, whose buffers were freed, and PyTorch raises "Trying to backward through the graph a second time". Passing `retain_graph=True` is not a fix: the graph then grows with every chunk, each backward pass re-traverses all earlier chunks, and it typically fails anyway because the optimizer has modified the weights in place. The opposite mistake, re-initialising the state to zeros every chunk, is also common: then the network never sees context from before the window at all, even in the forward pass. Reset the state only at true document boundaries, or when the batch columns are reshuffled.

> **Alternatives and relatives:** Real-time recurrent learning (Williams and Zipser, 1989) computes the exact online gradient but costs $O(H^4)$ per step for $H$ units; UORO (Tallec and Ollivier, 2018) gives a cheap unbiased but noisy approximation of it, and "Unbiasing truncated backpropagation through time" (Tallec and Ollivier, 2017) randomises $k_2$ and reweights to remove the bias in expectation. Memory-efficient BPTT (Gruslys et al., 2016) keeps full BPTT but recomputes activations instead of storing them. Transformer-XL (Dai et al., 2019) applies the same idea as TBPTT to attention: it caches the previous segment's hidden states as extra context with a stop-gradient.

**Strengths**

- Constant memory: $O(k_2)$ stored states for an arbitrarily long stream
- Frequent updates: one every $k_1$ steps
- Forward state still carries the whole past, so long context is usable at test time
- Trivial in PyTorch: `state = state.detach()` between chunks

**Limitations**

- Biased gradient: dependencies of $k_2$ or more steps are never learned
- $k_1<k_2$ costs $k_2/k_1$ times more backward (and re-forward) work per step
- Requires contiguous, unshuffled batches for the carried state to be meaningful
- Losses at the start of a window get little history unless windows overlap or vary in length

**Question:** A character-level LSTM is trained with TBPTT, k1 = k2 = 50, carrying (detached) states between chunks. Which statement is true?

- A. Its hidden state at test time can depend on characters more than 50 steps back, but it never received gradient to keep information that long on purpose
- B. It cannot use any information older than 50 characters, even at test time
- C. Its gradient is exact as long as the learning rate is small
- D. It is equivalent to full BPTT because the state is carried over

**Answer:** A. Its hidden state at test time can depend on characters more than 50 steps back, but it never received gradient to keep information that long on purpose — Detaching cuts only the backward graph. The forward state is passed on, so $h_t$ is a function of the whole past and can carry older information if it happens to persist. But no loss is ever connected by the graph to an input more than 49 links back, so there is no gradient signal for storing information for that purpose.

**Question:** In the figure's echo task (target = the input L steps ago), TBPTT(6, 6) learns lags 1–5. Which change would let it learn lag 7?

- A. Increase k2 to at least 8
- B. Decrease k1 to 1 while keeping k2 = 6
- C. Train for ten times as many updates
- D. Carry the hidden state between windows

**Answer:** A. Increase k2 to at least 8 — A lag-7 dependency needs $x_{t-7}$ inside the window of the loss at $t$, i.e. $k_2\ge8$ (the figure learns lag 7 with TBPTT(8, 8)). Lowering $k_1$ only makes windows overlap; with $k_2=6$ no loss is ever more than 5 links from its input. More updates cannot create a gradient that is identically zero, and the state is already carried.

**Key points**

- TBPTT$(k_1,k_2)$: every $k_1$ steps, backpropagate through the last $k_2$ steps; the state before the window is a constant (`detach()`).
- Memory $O(k_2)$ and an update every $k_1$ steps, instead of $O(T)$ memory and one update per sequence.
- The gradient is biased: terms older than the window are dropped, so dependencies of $k_2$ or more steps are never learned.
- Only the gradient is truncated; the forward state still carries the whole past.
- Stateful LM recipe: batchify into contiguous columns, chunk by `bptt`, detach between chunks, clip, never shuffle chunks; AWD-LSTM varies the window around 70 and rescales the learning rate.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
