Definition
Stacked RNN (also deep or multi-layer RNN) is a stack of recurrent layers in which layer takes the hidden-state sequence of layer as its input sequence: with . Each layer has its own weights and its own recurrent state; the top layer's states feed the output.
A single-layer RNN is already "deep in time": unrolled over steps it is a -layer network. But the computation between one step and the next, , is shallow: one affine map and one nonlinearity. Stacking gives every time step several layers of processing, so lower layers can track local, fast-changing features (characters, phonemes) and higher layers slower, more abstract ones (words, phrases). In practice two to four stacked layers usually beat one layer with the same number of parameters.
Hierarchies of recurrent nets go back to Schmidhuber's history compressor (1992) and El Hihi and Bengio's hierarchical RNNs (1996), but the modern case was made with LSTMs. Graves, Mohamed and Hinton (2013), Speech recognition with deep recurrent neural networks, stacked bidirectional LSTMs and reached 17.7% phoneme error on TIMIT, reporting that depth mattered more than layer width. Graves (2013), Generating sequences with recurrent neural networks, used stacks with skip connections from the input to every layer and from every layer to the output. Pascanu, Gulcehre, Cho and Bengio (2014), How to construct deep recurrent neural networks (ICLR), separated the places where depth can go: deep input-to-hidden, deep hidden-to-hidden transition, deep hidden-to-output, and the plain stack. Sutskever, Vinyals and Le (2014) used 4-layer LSTMs for translation and found each extra layer reduced perplexity by nearly 10%. Google's GNMT (Wu et al., 2016) went to 8 encoder and 8 decoder LSTM layers and needed residual connections to train them: they reported that plain stacks worked well up to 4 layers, barely with 6, and very poorly beyond 8.
Equations
- state of layer at step ; for LSTM/GRU layers replace the tanh update by the gated cell
- vertical (input) weights: for layer 1, above it (or if bidirectional)
- horizontal (recurrent) weights, one matrix per layer, shared over time
- readout from the top layer only (Graves 2013 instead connects every layer to the output)
A residual stack adds each layer's input to its output wherever the sizes match, as in GNMT: , where is what layer reads. The recurrence inside each layer is unchanged; only the vertical path gets an identity shortcut, exactly as in Residual neural network (ResNet). Inter-layer dropout (Zaremba, Sutskever and Vinyals, 2014) drops units on the vertical connections only, never on the recurrent ones, so the memory carried along is not corrupted at every step.
At a glance
- PyTorch
nn.RNN/GRU/LSTM(I, H, num_layers=L, dropout=p)- Parameters
- for layer 1 plus above it; for RNN, GRU, LSTM
- Compute
- linear in : times the per-step cost of one layer (when )
- `h_n` shape
- , layer-major;
h_n[-1]is the top layer - `out` shape
- with
batch_first=True: the top layer's states only - Typical depth
- 2–4 layers for language models and taggers; 8 with residuals in GNMT
- Dropout
dropout=pacts on the outputs of every layer except the last; withnum_layers=1it does nothing
Two kinds of depth
A stacked RNN is a grid: layers by steps. Information moves right (through time, via ) and up (through depth, via ). The path from input to the output at step is recurrent steps plus vertical steps, so stacking adds nonlinear steps to every path, including the shortest one. That is the source of both the extra capacity and the extra difficulty: the vertical Jacobians multiply just like the horizontal ones (see Vanilla RNN and Vanishing gradients).
Pascanu et al. (2014) argued that the plain stack makes the per-step transition no deeper: is still one layer. Their deep transition RNN inserts an MLP between and instead, which makes the recurrence itself deep but also lengthens every time path, so it needed shortcut connections to train. In practice the plain stack, with gated cells, won out because it is simple and maps onto fused kernels.
Parameter and compute scaling
For every layer costs the same, so parameters and multiply–adds per token grow exactly linearly in , and gates multiply both by . Bidirectional stacks double each layer and also double the input width of every layer above the first:
| module | parameters | MACs per token |
|---|---|---|
nn.LSTM(1024, 1024, num_layers=1) | 8,396,800 | 8.39 M |
nn.LSTM(1024, 1024, num_layers=4) | 33,587,200 | 33.55 M |
nn.RNN(1024, 1024, num_layers=8) | 16,793,600 | 16.78 M |
nn.GRU(1024, 1024, num_layers=8) | 50,380,800 | 50.33 M |
nn.LSTM(1024, 1024, num_layers=8) | 67,174,400 | 67.11 M |
nn.LSTM(1024, 1024, num_layers=8, bidirectional=True) | 193,069,056 | 192.94 M |
Wall-clock time grows more slowly than the MAC count when the wavefront is exploited, but memory for BPTT grows linearly in , since every layer's states at every step must be stored.
Worked example: a 2-layer BiLSTM tagger
nn.LSTM(input_size=300, hidden_size=512, num_layers=2, bidirectional=True, batch_first=True)
- 1Layer 1, each direction: . Two directions: .
- 2Layer 2 reads the concatenated features: per direction, for both.
- 3Total parameters. The second layer is almost twice the first because of the doubled input width.
- 4Shapes for a batch of sequences of length :
outis (top layer, forward and backward concatenated);h_nandc_nare , ordered layer 1 forward, layer 1 backward, layer 2 forward, layer 2 backward. - 5
h_n.view(2, 2, 32, 512)[-1]gives the top layer's two final states:[0]forward (after step 40),[1]backward (after reading back to step 1).
1import torch, torch.nn as nn23N, T, I, H, L = 32, 40, 300, 512, 24x = torch.randn(N, T, I)56# 1) Fused stack (cuDNN): dropout is applied to the outputs of layers 1..L-1 only7lstm = nn.LSTM(I, H, num_layers=L, dropout=0.3, bidirectional=True, batch_first=True)8out, (h_n, c_n) = lstm(x)9print(out.shape, h_n.shape) # torch.Size([32, 40, 1024]) torch.Size([4, 32, 512])10print(sum(p.numel() for p in lstm.parameters())) # 963379211top = h_n.view(L, 2, N, H)[-1] # (2, N, H): top layer, [0] forward, [1] backward12feat = torch.cat([top[0], top[1]], dim=-1) # (N, 2H) sentence vector1314# 2) Residual stack (GNMT style): nn.LSTM has no residual option, so stack 1-layer modules15class ResidualLSTM(nn.Module):16 def __init__(self, I, H, L, p=0.2):17 super().__init__()18 self.layers = nn.ModuleList(nn.LSTM(I if l == 0 else H, H, batch_first=True) for l in range(L))19 self.drop = nn.Dropout(p)20 def forward(self, x, state=None):21 state = state or [None] * len(self.layers)22 new_state = []23 for l, layer in enumerate(self.layers):24 y, s = layer(x, state[l])25 new_state.append(s)26 y = self.drop(y) if l < len(self.layers) - 1 else y # vertical dropout only27 x = y + x if l > 0 else y # residual where sizes match28 return x, new_state2930enc = ResidualLSTM(I, H, L=8)31y, states = enc(x)32print(y.shape, len(states), states[-1][0].shape) # torch.Size([32, 40, 512]) 8 torch.Size([1, 32, 512])Use when
- +Hierarchical features: each layer can work at a different time scale
- +2–4 layers usually beat one wider layer for the same parameter budget
- +Drop-in: one
num_layersargument, fused cuDNN kernels - +Wavefront parallelism and per-layer model parallelism (GNMT)
Avoid when
- −Parameters, compute and BPTT memory grow linearly in depth
- −Beyond about 4 layers plain stacks become hard to train; residuals are needed
- −Extra layers lengthen every path, including the shortest input-to-output one
- −Fused modules expose only the top layer's sequence and no residual option
Check yourself
You run nn.GRU(128, 256, num_layers=3, bidirectional=True, batch_first=True) on a batch of 16 sequences of length 50. What is the shape of h_n?
Check yourself
Why does a deep stack with residual connections between layers train better than a plain stack of the same depth?
Remember
- ✓Layer reads layer 's state sequence: , .
- ✓Depth adds per-step processing; 2–4 layers are typical (Graves et al. 2013, Sutskever et al. 2014).
- ✓Deeper stacks need residual connections between layers, as in GNMT's 8 + 8 LSTM layers.
- ✓Parameters and MACs grow linearly in ;
h_nis andoutis only the top layer. - ✓Use
dropout=pfor vertical dropout between layers; it does nothing with one layer.