Stacked RNN

Recurrent layers stacked in depth: layer l reads the hidden sequence of layer l − 1 at the same step. Two to four layers usually beat one wide layer; deeper stacks need residual connections (GNMT used 8), and parameters and compute grow linearly with depth.

12 min read1013 wordscore0 viewsupdated 2026-10-12

Definition

Stacked RNN (also deep or multi-layer RNN) is a stack of LL recurrent layers in which layer ll takes the hidden-state sequence of layer l−1l-1 as its input sequence: ht(l)=f(l)(ht−1(l), ht(l−1))h^{(l)}_t=f^{(l)}\big(h^{(l)}_{t-1},\,h^{(l-1)}_t\big) with ht(0)=xth^{(0)}_t=x_t. Each layer has its own weights and its own recurrent state; the top layer's states feed the output.

A single-layer RNN is already "deep in time": unrolled over TT steps it is a TT-layer network. But the computation between one step and the next, ht=tanh⁡(Wxhxt+Whhht−1+b)h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b), is shallow: one affine map and one nonlinearity. Stacking gives every time step several layers of processing, so lower layers can track local, fast-changing features (characters, phonemes) and higher layers slower, more abstract ones (words, phrases). In practice two to four stacked layers usually beat one layer with the same number of parameters.

Hierarchies of recurrent nets go back to Schmidhuber's history compressor (1992) and El Hihi and Bengio's hierarchical RNNs (1996), but the modern case was made with LSTMs. Graves, Mohamed and Hinton (2013), Speech recognition with deep recurrent neural networks, stacked bidirectional LSTMs and reached 17.7% phoneme error on TIMIT, reporting that depth mattered more than layer width. Graves (2013), Generating sequences with recurrent neural networks, used stacks with skip connections from the input to every layer and from every layer to the output. Pascanu, Gulcehre, Cho and Bengio (2014), How to construct deep recurrent neural networks (ICLR), separated the places where depth can go: deep input-to-hidden, deep hidden-to-hidden transition, deep hidden-to-output, and the plain stack. Sutskever, Vinyals and Le (2014) used 4-layer LSTMs for translation and found each extra layer reduced perplexity by nearly 10%. Google's GNMT (Wu et al., 2016) went to 8 encoder and 8 decoder LSTM layers and needed residual connections to train them: they reported that plain stacks worked well up to 4 layers, barely with 6, and very poorly beyond 8.

Equations

ht(l)=tanh⁡ ⁣(W(l)ht(l−1)+U(l)ht−1(l)+b(l)),l=1,…,L,ht(0)=xt,yt=g(ht(L))h^{(l)}_t=\tanh\!\big(W^{(l)}h^{(l-1)}_t+U^{(l)}h^{(l)}_{t-1}+b^{(l)}\big),\quad l=1,\dots,L,\qquad h^{(0)}_t=x_t,\qquad y_t=g\big(h^{(L)}_t\big)
ht(l)h^{(l)}_t
state of layer ll at step tt; for LSTM/GRU layers replace the tanh update by the gated cell
W(l)W^{(l)}
vertical (input) weights: H×IH\times I for layer 1, H×HH\times H above it (or H×2HH\times 2H if bidirectional)
U(l)U^{(l)}
horizontal (recurrent) weights, one matrix per layer, shared over time
gg
readout from the top layer only (Graves 2013 instead connects every layer to the output)

A residual stack adds each layer's input to its output wherever the sizes match, as in GNMT: xt(l+1)=ht(l)+xt(l)x^{(l+1)}_t=h^{(l)}_t+x^{(l)}_t, where xt(l)x^{(l)}_t is what layer ll reads. The recurrence inside each layer is unchanged; only the vertical path gets an identity shortcut, exactly as in Residual neural network (ResNet). Inter-layer dropout (Zaremba, Sutskever and Vinyals, 2014) drops units on the vertical connections only, never on the recurrent ones, so the memory carried along tt is not corrupted at every step.

At a glance

PyTorch
nn.RNN/GRU/LSTM(I, H, num_layers=L, dropout=p)
Parameters
G H (I+H+2)G\,H\,(I+H+2) for layer 1 plus (L−1) G H (H+H+2)(L-1)\,G\,H\,(H+H+2) above it; G=1,3,4G=1,3,4 for RNN, GRU, LSTM
Compute
linear in LL: LL times the per-step cost of one layer (when I=HI=H)
`h_n` shape
(L⋅dirs, N, H)(L\cdot\text{dirs},\,N,\,H), layer-major; h_n[-1] is the top layer
`out` shape
(N,T,dirs⋅H)(N,T,\text{dirs}\cdot H) with batch_first=True: the top layer's states only
Typical depth
2–4 layers for language models and taggers; 8 with residuals in GNMT
Dropout
dropout=p acts on the outputs of every layer except the last; with num_layers=1 it does nothing
‖∂y/∂hₜˡ‖₂, y = top output at t = 12 (log₁₀ in each cell)layer 1layer 2layer 3layer 4input xₜ−0.8−0.8−0.8−0.7−0.5−0.4−0.4−0.3−0.2−0.2−0.2−0.3−0.8−0.7−0.7−0.6−0.5−0.5−0.4−0.3−0.2−0.1−0.1−0.0−0.9−0.9−0.9−0.7−0.6−0.5−0.3−0.3−0.20.00.00.0−0.9−0.8−0.7−0.6−0.6−0.5−0.4−0.2−0.2−0.1−0.00.0−0.9−0.9−0.8−0.7−0.7−0.6−0.3−0.6−0.3−0.2−0.8−1.0t=1t=4t=8t=12‖∂y/∂x₁₂‖ = 0.094 (up 4 layers) · ‖∂y/∂x₁‖ = 0.123 (11 steps back)toy grid: 8 tanh units per layer, W_hh singular values 0.9, gain 0.70plain stack: no skip connections · wavefront: T + L − 1 stepsPyTorch sizes (any I = H)nn.LSTM(1024, 1024,num_layers=4)parameters33,587,200MACs per token: 33.55 Mout (batch_first):(N, T, 1024)h_n and c_n:(4, N, 1024) = (L·dirs, N, H)h_n[-1]: top layer, last steplayer 1: 4·H·(I+H+2) per dirlayer ≥2: 4·H·(H+H+2) per dir■ strong ■ ≤ 1e−3 ■ > 1 ( residual skip □ output cell
cellH
fig.Left: a toy stack of tanh layers (8 units each, every W_hh with singular values 0.9) over 12 steps. Each cell shows log₁₀‖∂y/∂hₜˡ‖, the sensitivity of the top output at the last step y to that state; the circles are the raw inputs. With the default 4 plain layers and vertical gain 0.70 the input at the same step reaches y with ‖∂y/∂x₁₂‖ = 0.094. Raise num_layers to 8 and it drops to 0.013 (to 0.0025 at gain 0.5): every extra layer multiplies in another tanh Jacobian. Switch residual on and the same 8-layer stack gives 1.95 (1.37 at gain 0.5), because the identity skips carry the gradient straight down. Right: parameter count, multiply–adds per token and tensor shapes of the matching PyTorch module; nn.LSTM(1024, 1024, num_layers=8), GNMT's layer size, has 67,174,400 parameters.

Two kinds of depth

A stacked RNN is a grid: LL layers by TT steps. Information moves right (through time, via U(l)U^{(l)}) and up (through depth, via W(l)W^{(l)}). The path from input xtx_t to the output at step t′t' is t′−tt'-t recurrent steps plus LL vertical steps, so stacking adds L−1L-1 nonlinear steps to every path, including the shortest one. That is the source of both the extra capacity and the extra difficulty: the vertical Jacobians multiply just like the horizontal ones (see Vanilla RNN and Vanishing gradients).

Pascanu et al. (2014) argued that the plain stack makes the per-step transition no deeper: ht−1(l)→ht(l)h^{(l)}_{t-1}\to h^{(l)}_t is still one layer. Their deep transition RNN inserts an MLP between ht−1h_{t-1} and hth_t instead, which makes the recurrence itself deep but also lengthens every time path, so it needed shortcut connections to train. In practice the plain stack, with gated cells, won out because it is simple and maps onto fused kernels.

Parameter and compute scaling

For I=HI=H every layer costs the same, so parameters and multiply–adds per token grow exactly linearly in LL, and gates multiply both by GG. Bidirectional stacks double each layer and also double the input width of every layer above the first:

moduleparametersMACs per token
nn.LSTM(1024, 1024, num_layers=1)8,396,8008.39 M
nn.LSTM(1024, 1024, num_layers=4)33,587,20033.55 M
nn.RNN(1024, 1024, num_layers=8)16,793,60016.78 M
nn.GRU(1024, 1024, num_layers=8)50,380,80050.33 M
nn.LSTM(1024, 1024, num_layers=8)67,174,40067.11 M
nn.LSTM(1024, 1024, num_layers=8, bidirectional=True)193,069,056192.94 M
Sizes with I = H, both bias vectors counted (computed by the figure's code)

Wall-clock time grows more slowly than the MAC count when the wavefront is exploited, but memory for BPTT grows linearly in L⋅TL\cdot T, since every layer's states at every step must be stored.

Worked example: a 2-layer BiLSTM tagger

nn.LSTM(input_size=300, hidden_size=512, num_layers=2, bidirectional=True, batch_first=True)

  1. 1Layer 1, each direction: 4⋅512⋅(300+512+2)=1,667,0724\cdot512\cdot(300+512+2)=1{,}667{,}072. Two directions: 3,334,1443{,}334{,}144.
  2. 2Layer 2 reads the concatenated 2⋅512=10242\cdot512=1024 features: 4⋅512⋅(1024+512+2)=3,149,8244\cdot512\cdot(1024+512+2)=3{,}149{,}824 per direction, 6,299,6486{,}299{,}648 for both.
  3. 3Total 9,633,7929{,}633{,}792 parameters. The second layer is almost twice the first because of the doubled input width.
  4. 4Shapes for a batch of N=32N=32 sequences of length T=40T=40: out is (32,40,1024)(32,40,1024) (top layer, forward and backward concatenated); h_n and c_n are (4,32,512)(4,32,512), ordered layer 1 forward, layer 1 backward, layer 2 forward, layer 2 backward.
  5. 5h_n.view(2, 2, 32, 512)[-1] gives the top layer's two final states: [0] forward (after step 40), [1] backward (after reading back to step 1).
python
1import torch, torch.nn as nn
2
3N, T, I, H, L = 32, 40, 300, 512, 2
4x = torch.randn(N, T, I)
5
6# 1) Fused stack (cuDNN): dropout is applied to the outputs of layers 1..L-1 only
7lstm = nn.LSTM(I, H, num_layers=L, dropout=0.3, bidirectional=True, batch_first=True)
8out, (h_n, c_n) = lstm(x)
9print(out.shape, h_n.shape) # torch.Size([32, 40, 1024]) torch.Size([4, 32, 512])
10print(sum(p.numel() for p in lstm.parameters())) # 9633792
11top = h_n.view(L, 2, N, H)[-1] # (2, N, H): top layer, [0] forward, [1] backward
12feat = torch.cat([top[0], top[1]], dim=-1) # (N, 2H) sentence vector
13
14# 2) Residual stack (GNMT style): nn.LSTM has no residual option, so stack 1-layer modules
15class ResidualLSTM(nn.Module):
16 def __init__(self, I, H, L, p=0.2):
17 super().__init__()
18 self.layers = nn.ModuleList(nn.LSTM(I if l == 0 else H, H, batch_first=True) for l in range(L))
19 self.drop = nn.Dropout(p)
20 def forward(self, x, state=None):
21 state = state or [None] * len(self.layers)
22 new_state = []
23 for l, layer in enumerate(self.layers):
24 y, s = layer(x, state[l])
25 new_state.append(s)
26 y = self.drop(y) if l < len(self.layers) - 1 else y # vertical dropout only
27 x = y + x if l > 0 else y # residual where sizes match
28 return x, new_state
29
30enc = ResidualLSTM(I, H, L=8)
31y, states = enc(x)
32print(y.shape, len(states), states[-1][0].shape) # torch.Size([32, 40, 512]) 8 torch.Size([1, 32, 512])
Fused stack with inter-layer dropout, h_n indexing, and a residual stack built from single layers

Use when

  • +Hierarchical features: each layer can work at a different time scale
  • +2–4 layers usually beat one wider layer for the same parameter budget
  • +Drop-in: one num_layers argument, fused cuDNN kernels
  • +Wavefront parallelism and per-layer model parallelism (GNMT)

Avoid when

  • −Parameters, compute and BPTT memory grow linearly in depth
  • −Beyond about 4 layers plain stacks become hard to train; residuals are needed
  • −Extra layers lengthen every path, including the shortest input-to-output one
  • −Fused modules expose only the top layer's sequence and no residual option

Check yourself

You run nn.GRU(128, 256, num_layers=3, bidirectional=True, batch_first=True) on a batch of 16 sequences of length 50. What is the shape of h_n?

Check yourself

Why does a deep stack with residual connections between layers train better than a plain stack of the same depth?

Remember

  • ✓Layer ll reads layer l−1l-1's state sequence: ht(l)=f(ht−1(l),ht(l−1))h^{(l)}_t=f(h^{(l)}_{t-1},h^{(l-1)}_t), ht(0)=xth^{(0)}_t=x_t.
  • ✓Depth adds per-step processing; 2–4 layers are typical (Graves et al. 2013, Sutskever et al. 2014).
  • ✓Deeper stacks need residual connections between layers, as in GNMT's 8 + 8 LSTM layers.
  • ✓Parameters and MACs grow linearly in LL; h_n is (L⋅dirs,N,H)(L\cdot\text{dirs},N,H) and out is only the top layer.
  • ✓Use dropout=p for vertical dropout between layers; it does nothing with one layer.
← back to Deep Learning roadmap