# Stacked RNN

> Recurrent layers stacked in depth: layer l reads the hidden sequence of layer l − 1 at the same step. Two to four layers usually beat one wide layer; deeper stacks need residual connections (GNMT used 8), and parameters and compute grow linearly with depth.

Source: https://www.mlfoundry.in/learn/stacked-rnn  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 12 min read · updated 2026-10-12

**Stacked RNN:** (also **deep** or **multi-layer RNN**) is a stack of $L$ recurrent layers in which layer $l$ takes the hidden-state sequence of layer $l-1$ as its input sequence: $h^{(l)}_t=f^{(l)}\big(h^{(l)}_{t-1},\,h^{(l-1)}_t\big)$ with $h^{(0)}_t=x_t$. Each layer has its own weights and its own recurrent state; the top layer's states feed the output.

A single-layer RNN is already "deep in time": unrolled over $T$ steps it is a $T$-layer network. But the computation *between* one step and the next, $h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b)$, is shallow: one affine map and one nonlinearity. Stacking gives every time step several layers of processing, so lower layers can track local, fast-changing features (characters, phonemes) and higher layers slower, more abstract ones (words, phrases). In practice two to four stacked layers usually beat one layer with the same number of parameters.

Hierarchies of recurrent nets go back to Schmidhuber's history compressor (1992) and El Hihi and Bengio's hierarchical RNNs (1996), but the modern case was made with LSTMs. **Graves, Mohamed and Hinton (2013)**, *Speech recognition with deep recurrent neural networks*, stacked bidirectional LSTMs and reached 17.7% phoneme error on TIMIT, reporting that depth mattered more than layer width. **Graves (2013)**, *Generating sequences with recurrent neural networks*, used stacks with skip connections from the input to every layer and from every layer to the output. **Pascanu, Gulcehre, Cho and Bengio (2014)**, *How to construct deep recurrent neural networks* (ICLR), separated the places where depth can go: deep input-to-hidden, deep hidden-to-hidden *transition*, deep hidden-to-output, and the plain stack. **Sutskever, Vinyals and Le (2014)** used 4-layer LSTMs for translation and found each extra layer reduced perplexity by nearly 10%. Google's **GNMT** (Wu et al., 2016) went to 8 encoder and 8 decoder LSTM layers and needed **residual connections** to train them: they reported that plain stacks worked well up to 4 layers, barely with 6, and very poorly beyond 8.

## Equations

$$
h^{(l)}_t=\tanh\!\big(W^{(l)}h^{(l-1)}_t+U^{(l)}h^{(l)}_{t-1}+b^{(l)}\big),\quad l=1,\dots,L,\qquad h^{(0)}_t=x_t,\qquad y_t=g\big(h^{(L)}_t\big)
$$

- $h^{(l)}_t$: state of layer $l$ at step $t$; for LSTM/GRU layers replace the tanh update by the gated cell
- $W^{(l)}$: vertical (input) weights: $H\times I$ for layer 1, $H\times H$ above it (or $H\times 2H$ if bidirectional)
- $U^{(l)}$: horizontal (recurrent) weights, one matrix per layer, shared over time
- $g$: readout from the **top** layer only (Graves 2013 instead connects every layer to the output)

A **residual** stack adds each layer's input to its output wherever the sizes match, as in GNMT: $x^{(l+1)}_t=h^{(l)}_t+x^{(l)}_t$, where $x^{(l)}_t$ is what layer $l$ reads. The recurrence inside each layer is unchanged; only the vertical path gets an identity shortcut, exactly as in **Residual neural network (ResNet)**. **Inter-layer dropout** (Zaremba, Sutskever and Vinyals, 2014) drops units on the vertical connections only, never on the recurrent ones, so the memory carried along $t$ is not corrupted at every step.

**At a glance**

- **PyTorch:** `nn.RNN/GRU/LSTM(I, H, num_layers=L, dropout=p)`
- **Parameters:** $G\,H\,(I+H+2)$ for layer 1 plus $(L-1)\,G\,H\,(H+H+2)$ above it; $G=1,3,4$ for RNN, GRU, LSTM
- **Compute:** linear in $L$: $L$ times the per-step cost of one layer (when $I=H$)
- **`h_n` shape:** $(L\cdot\text{dirs},\,N,\,H)$, layer-major; `h_n[-1]` is the top layer
- **`out` shape:** $(N,T,\text{dirs}\cdot H)$ with `batch_first=True`: the **top** layer's states only
- **Typical depth:** 2–4 layers for language models and taggers; 8 with residuals in GNMT
- **Dropout:** `dropout=p` acts on the outputs of every layer except the last; with `num_layers=1` it does nothing

> Figure: Left: a toy stack of tanh layers (8 units each, every W_hh with singular values 0.9) over 12 steps. Each cell shows log₁₀‖∂y/∂hₜˡ‖, the sensitivity of the top output at the last step y to that state; the circles are the raw inputs. With the default 4 plain layers and vertical gain 0.70 the input at the same step reaches y with ‖∂y/∂x₁₂‖ = 0.094. Raise num_layers to 8 and it drops to 0.013 (to 0.0025 at gain 0.5): every extra layer multiplies in another tanh Jacobian. Switch residual on and the same 8-layer stack gives 1.95 (1.37 at gain 0.5), because the identity skips carry the gradient straight down. Right: parameter count, multiply–adds per token and tensor shapes of the matching PyTorch module; nn.LSTM(1024, 1024, num_layers=8), GNMT's layer size, has 67,174,400 parameters.

## Two kinds of depth

A stacked RNN is a grid: $L$ layers by $T$ steps. Information moves right (through time, via $U^{(l)}$) and up (through depth, via $W^{(l)}$). The path from input $x_t$ to the output at step $t'$ is $t'-t$ recurrent steps plus $L$ vertical steps, so stacking adds $L-1$ nonlinear steps to **every** path, including the shortest one. That is the source of both the extra capacity and the extra difficulty: the vertical Jacobians multiply just like the horizontal ones (see **Vanilla RNN** and **Vanishing gradients**).

Pascanu et al. (2014) argued that the plain stack makes the per-step transition no deeper: $h^{(l)}_{t-1}\to h^{(l)}_t$ is still one layer. Their **deep transition** RNN inserts an MLP between $h_{t-1}$ and $h_t$ instead, which makes the recurrence itself deep but also lengthens every time path, so it needed shortcut connections to train. In practice the plain stack, with gated cells, won out because it is simple and maps onto fused kernels.

> **Wavefront parallelism:** Cell $(l,t)$ needs only $(l-1,t)$ and $(l,t-1)$. All cells on an anti-diagonal $l+t=\text{const}$ are independent, so a stack can run as a wavefront in $T+L-1$ sequential steps instead of $T\cdot L$. GNMT exploited this by placing each layer on its own GPU: layer $l$ starts step $t$ as soon as layer $l-1$ has finished it. Making only the bottom encoder layer bidirectional kept that pipeline intact, because a backward layer has to wait for the whole sequence.

## Parameter and compute scaling

For $I=H$ every layer costs the same, so parameters and multiply–adds per token grow exactly linearly in $L$, and gates multiply both by $G$. Bidirectional stacks double each layer and also double the input width of every layer above the first:

Sizes with I = H, both bias vectors counted (computed by the figure's code)

| module | parameters | MACs per token |
| --- | --- | --- |
| `nn.LSTM(1024, 1024, num_layers=1)` | 8,396,800 | 8.39 M |
| `nn.LSTM(1024, 1024, num_layers=4)` | 33,587,200 | 33.55 M |
| `nn.RNN(1024, 1024, num_layers=8)` | 16,793,600 | 16.78 M |
| `nn.GRU(1024, 1024, num_layers=8)` | 50,380,800 | 50.33 M |
| `nn.LSTM(1024, 1024, num_layers=8)` | 67,174,400 | 67.11 M |
| `nn.LSTM(1024, 1024, num_layers=8, bidirectional=True)` | 193,069,056 | 192.94 M |

Wall-clock time grows more slowly than the MAC count when the wavefront is exploited, but memory for BPTT grows linearly in $L\cdot T$, since every layer's states at every step must be stored.

## Worked example: a 2-layer BiLSTM tagger

**nn.LSTM(input_size=300, hidden_size=512, num_layers=2, bidirectional=True, batch_first=True)**

1. Layer 1, each direction: $4\cdot512\cdot(300+512+2)=1{,}667{,}072$. Two directions: $3{,}334{,}144$.
2. Layer 2 reads the concatenated $2\cdot512=1024$ features: $4\cdot512\cdot(1024+512+2)=3{,}149{,}824$ per direction, $6{,}299{,}648$ for both.
3. Total $9{,}633{,}792$ parameters. The second layer is almost twice the first because of the doubled input width.
4. Shapes for a batch of $N=32$ sequences of length $T=40$: `out` is $(32,40,1024)$ (top layer, forward and backward concatenated); `h_n` and `c_n` are $(4,32,512)$, ordered layer 1 forward, layer 1 backward, layer 2 forward, layer 2 backward.
5. `h_n.view(2, 2, 32, 512)[-1]` gives the top layer's two final states: `[0]` forward (after step 40), `[1]` backward (after reading back to step 1).

```python
import torch, torch.nn as nn

N, T, I, H, L = 32, 40, 300, 512, 2
x = torch.randn(N, T, I)

# 1) Fused stack (cuDNN): dropout is applied to the outputs of layers 1..L-1 only
lstm = nn.LSTM(I, H, num_layers=L, dropout=0.3, bidirectional=True, batch_first=True)
out, (h_n, c_n) = lstm(x)
print(out.shape, h_n.shape)          # torch.Size([32, 40, 1024]) torch.Size([4, 32, 512])
print(sum(p.numel() for p in lstm.parameters()))   # 9633792
top = h_n.view(L, 2, N, H)[-1]       # (2, N, H): top layer, [0] forward, [1] backward
feat = torch.cat([top[0], top[1]], dim=-1)          # (N, 2H) sentence vector

# 2) Residual stack (GNMT style): nn.LSTM has no residual option, so stack 1-layer modules
class ResidualLSTM(nn.Module):
    def __init__(self, I, H, L, p=0.2):
        super().__init__()
        self.layers = nn.ModuleList(nn.LSTM(I if l == 0 else H, H, batch_first=True) for l in range(L))
        self.drop = nn.Dropout(p)
    def forward(self, x, state=None):
        state = state or [None] * len(self.layers)
        new_state = []
        for l, layer in enumerate(self.layers):
            y, s = layer(x, state[l])
            new_state.append(s)
            y = self.drop(y) if l < len(self.layers) - 1 else y   # vertical dropout only
            x = y + x if l > 0 else y                            # residual where sizes match
        return x, new_state

enc = ResidualLSTM(I, H, L=8)
y, states = enc(x)
print(y.shape, len(states), states[-1][0].shape)    # torch.Size([32, 40, 512]) 8 torch.Size([1, 32, 512])
```

Fused stack with inter-layer dropout, h_n indexing, and a residual stack built from single layers

> **out is not every layer:** `out` holds only the **top** layer's states. If you want features from intermediate layers (ELMo mixes all layers of a biLSTM, see **Bidirectional LSTM**), build the stack from single-layer modules as in the code, or run the layers one at a time. Also remember `h_n[-1]` in a bidirectional stack is the top layer's **backward** state; the top forward state is `h_n[-2]`.

> **dropout with num_layers=1:** `nn.LSTM(..., num_layers=1, dropout=0.5)` applies no dropout at all (PyTorch warns), because there is no layer below the last. Put an `nn.Dropout` on the output yourself. Dropout on the recurrent connections needs a different scheme: variational dropout (Gal and Ghahramani, 2016) or weight-dropped LSTMs (see **DropConnect**).

> **Related designs:** Bidirectional stacks are covered in **Bidirectional RNN** and **Bidirectional LSTM**; stacked encoders and decoders in **Sequence-to-sequence (Seq2Seq)**. Gated cells themselves are in **Long short-term memory (LSTM)** and **Gated recurrent unit (GRU)**. Transformers stack layers the same way, with residuals and layer normalisation around each block (see **Self-Attention** and **Layer normalization**).

**Strengths**

- Hierarchical features: each layer can work at a different time scale
- 2–4 layers usually beat one wider layer for the same parameter budget
- Drop-in: one `num_layers` argument, fused cuDNN kernels
- Wavefront parallelism and per-layer model parallelism (GNMT)

**Limitations**

- Parameters, compute and BPTT memory grow linearly in depth
- Beyond about 4 layers plain stacks become hard to train; residuals are needed
- Extra layers lengthen every path, including the shortest input-to-output one
- Fused modules expose only the top layer's sequence and no residual option

**Question:** You run nn.GRU(128, 256, num_layers=3, bidirectional=True, batch_first=True) on a batch of 16 sequences of length 50. What is the shape of h_n?

- A. (6, 16, 256)
- B. (16, 6, 256)
- C. (3, 16, 512)
- D. (16, 50, 512)

**Answer:** A. (6, 16, 256) — `h_n` is $(L\cdot\text{dirs},N,H)=(3\cdot2,16,256)$, and `batch_first` does not affect it. $(16,50,512)$ is `out`, the top layer's forward and backward states concatenated at every step.

**Question:** Why does a deep stack with residual connections between layers train better than a plain stack of the same depth?

- A. The identity shortcut adds a path whose vertical Jacobian is I, so the gradient no longer shrinks with every layer
- B. Residuals reduce the parameter count
- C. Residuals remove the recurrence inside each layer
- D. Residuals make the layers bidirectional

**Answer:** A. The identity shortcut adds a path whose vertical Jacobian is I, so the gradient no longer shrinks with every layer — With $x^{(l+1)}=h^{(l)}+x^{(l)}$, $\partial x^{(l+1)}/\partial x^{(l)}=I+\partial h^{(l)}/\partial x^{(l)}$. The product over layers always contains the identity term, so the gradient at the bottom does not decay geometrically with $L$. In the figure, 8 plain layers give 0.013 and 8 residual layers 1.95.

**Key points**

- Layer $l$ reads layer $l-1$'s state sequence: $h^{(l)}_t=f(h^{(l)}_{t-1},h^{(l-1)}_t)$, $h^{(0)}_t=x_t$.
- Depth adds per-step processing; 2–4 layers are typical (Graves et al. 2013, Sutskever et al. 2014).
- Deeper stacks need residual connections between layers, as in GNMT's 8 + 8 LSTM layers.
- Parameters and MACs grow linearly in $L$; `h_n` is $(L\cdot\text{dirs},N,H)$ and `out` is only the top layer.
- Use `dropout=p` for vertical dropout between layers; it does nothing with one layer.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
