Definition
Vanilla RNN (also Elman network or simple RNN) is the recurrent cell : the new hidden state is a squashed affine function of the current input and the previous state. An output layer reads the state. There are no gates; the whole memory mechanism is the matrix .
This is the cell that Recurrent neural network (RNN) describes in general terms, made concrete. Elman (1990) introduced it as a feed-forward net whose hidden layer is copied into "context units" and fed back at the next step. It is the baseline every gated cell is compared against, and it is still worth understanding in detail, because its failure mode, the product of Jacobians across time, is exactly what LSTM, GRU, orthogonal initialisation and gradient clipping were designed to fix.
Two papers frame its training. Bengio, Simard and Frasconi (1994) showed that learning long-range dependencies with gradient descent is hard in general. Pascanu, Mikolov and Bengio (2013), On the difficulty of training recurrent neural networks (ICML), made the analysis precise for this cell: they bounded the BPTT Jacobians by the largest singular value of , explained exploding gradients as "cliffs" in the loss surface, and proposed rescaling the gradient norm (clipping) as the remedy. Karpathy's min-char-rnn (2015), about a hundred lines of NumPy implementing exactly this cell on characters, made it the standard teaching example.
The cell
- input at step (a one-hot character or an embedding)
- hidden state, unless given
- input-to-hidden weights (PyTorch: weight_ih_l0)
- hidden-to-hidden (recurrent) weights (weight_hh_l0)
- bias; PyTorch keeps two, , for cuDNN compatibility
- readout to classes, usually a separate
nn.Linear
Counting parameters: has entries, has , and the two bias vectors , so nn.RNN(I, H) has parameters. A character model with symbols (tiny Shakespeare) and has in the cell plus in the readout. The recurrent matrix dominates once . Compute is two matrix–vector products per step, , and the steps must run one after another.
At a glance
- Update
- Parameters
- in PyTorch (two biases); with one
- Local Jacobian
- Vanishing (sufficient)
- with tanh (Pascanu et al., 2013)
- Exploding (necessary)
- Practical reach
- dependencies beyond a few tens of steps are hard to learn
- PyTorch
nn.RNN,nn.RNNCell,nonlinearity="tanh"or"relu"
The full BPTT gradient
Let be the error at the pre-activation of step . Backpropagation through time runs the recurrence backwards. Each step receives the loss's direct gradient through the readout and the gradient coming back from step :
- softmax-plus-cross-entropy gradient at step t (one-hot target)
- the error flowing back in time from the next step
- : near 0 when a unit is saturated at
- nothing comes back from beyond the last step
Unrolling the recursion, the error that the loss at step sends back to step passes through a product of local Jacobians:
- largest singular value of the recurrent matrix (for a normal matrix, equal to the spectral radius )
- bound on : 1 for tanh, 1/4 for the logistic sigmoid
- the time lag: the bound is exponential in it
Pascanu et al. (2013) read two conditions off this bound. If , every factor is a contraction and long-term contributions vanish exponentially; this is sufficient. If gradients explode, then must hold; this is only necessary. The asymmetry matters. A recurrent matrix with large singular values usually drives the units into saturation, where , and the diagonal factors then kill the product. Exploding happens in a narrower regime: expanding, but the states still in tanh's linear range. For a linear RNN the picture is exact: , which grows or shrinks like along the leading eigenvector.
Taming the gradient
Exploding gradients are rare but catastrophic: one step on a cliff throws the weights far away and the loss spikes or turns into NaN. The standard fix is gradient norm clipping (Pascanu et al., 2013): compute the norm of the full gradient vector and, if it exceeds a threshold , rescale it to length while keeping its direction. Mikolov had earlier clipped element-wise in his RNN language models. See Gradient clipping for the full story.
- the concatenated gradient of all parameters
- threshold, e.g. 1–5 for RNN language models; set it slightly above the typical norm
Vanishing gradients have no such one-line fix, because a tiny gradient is not wrong, just uninformative. The remedies change the model. Initialise orthogonal (all singular values 1; Saxe et al., 2014) or as the identity with ReLU units (IRNN; Le, Jaitly and Hinton, 2015). Constrain to be unitary (Arjovsky et al., 2016). Add a soft penalty that keeps near 1 (Pascanu et al.'s regulariser). Or, most commonly, replace the cell with an additive, gated one: Long short-term memory (LSTM) or Gated recurrent unit (GRU).
Worked example: a 2-unit character model on "hell" → "ello"
Vocabulary , one-hot inputs, , all biases 0. The columns of are the characters' input vectors , , , . The recurrent matrix is (singular values 0.723 and 0.581, spectral radius 0.648), and the readout rows are , , , . The model reads "hell" and should predict the next character at each step: "ello".
| t | input → target | a_t | h_t | p(target) | −log p | argmax |
|---|---|---|---|---|---|---|
| 1 | h → e | (1.0000, 0.0000) | (0.7616, 0.0000) | 0.0982 | 2.3209 | h |
| 2 | e → l | (0.3808, 1.2285) | (0.3634, 0.8421) | 0.4343 | 0.8340 | l |
| 3 | l → l | (−1.1552, 1.1143) | (−0.8195, 0.8056) | 0.1444 | 1.9350 | e |
| 4 | l → o | (−1.7320, 0.7375) | (−0.9393, 0.6276) | 0.1283 | 2.0532 | e |
Reading the example
- 1Step 2: , so .
- 2Logits ; the softmax gives , the one correct prediction so far.
- 3Steps 3 and 4 both read "l" but need different answers (l, then o). Only the hidden state can tell them apart: and differ because . Training must shape so that this difference is enough.
- 4Mean loss nats, worse than the uniform guess : the weights are untrained.
- 5Backward in time: , and over three steps , already a 27-fold shrink with .
1import torch, torch.nn as nn, torch.nn.functional as F23# --- shapes ---------------------------------------------------------------4rnn = nn.RNN(input_size=4, hidden_size=2, nonlinearity="tanh", batch_first=True)5x = torch.randn(3, 5, 4) # (N, T, I) because batch_first=True6h0 = torch.zeros(1, 3, 2) # (num_layers * num_directions, N, H) -- NOT batch-first7out, h_n = rnn(x, h0)8print(out.shape, h_n.shape) # torch.Size([3, 5, 2]) torch.Size([1, 3, 2])9assert torch.allclose(out[:, -1], h_n[0])10print(sum(p.numel() for p in rnn.parameters())) # 2*(4+2+2) = 161112# --- the same cell by hand ---------------------------------------------------13W_ih, W_hh, b_ih, b_hh = rnn.weight_ih_l0, rnn.weight_hh_l0, rnn.bias_ih_l0, rnn.bias_hh_l014h = h0[0]15for t in range(x.shape[1]):16 h = torch.tanh(x[:, t] @ W_ih.T + b_ih + h @ W_hh.T + b_hh)17assert torch.allclose(h, h_n[0], atol=1e-6)1819# --- char-level language model ------------------------------------------------20text = "hello world, hello rnn. " * 5021vocab = sorted(set(text)); stoi = {c: i for i, c in enumerate(vocab)}22data = torch.tensor([stoi[c] for c in text])23V, H, T = len(vocab), 128, 322425class CharRNN(nn.Module):26 def __init__(self):27 super().__init__()28 self.rnn = nn.RNN(V, H, batch_first=True)29 self.head = nn.Linear(H, V)30 def forward(self, idx, h=None):31 out, h = self.rnn(F.one_hot(idx, V).float(), h)32 return self.head(out), h3334model = CharRNN()35nn.init.orthogonal_(model.rnn.weight_hh_l0) # all singular values 1 at the start36opt = torch.optim.Adam(model.parameters(), lr=3e-3)37for step in range(300):38 i = torch.randint(0, len(data) - T - 1, (16,))39 xb = torch.stack([data[j:j + T] for j in i]); yb = torch.stack([data[j + 1:j + T + 1] for j in i])40 logits, _ = model(xb)41 loss = F.cross_entropy(logits.reshape(-1, V), yb.reshape(-1))42 opt.zero_grad(); loss.backward()43 norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # returns the pre-clip norm44 opt.step()Use when
- +Minimal: one affine map and a tanh, parameters
- +Cheapest recurrent cell per step; easy to analyse exactly
- +Fine for short dependencies (tens of steps) and as a baseline
- +Saturating tanh keeps states bounded in
Avoid when
- −Long-range gradients vanish whenever , and saturation makes them vanish even above 1
- −Exploding gradients on loss cliffs: clipping is mandatory
- −Sensitive to the initialisation of the recurrent matrix
- −Outperformed by LSTM and GRU on almost every task with long-range structure
Check yourself
The largest singular value of a tanh RNN's W_hh is 0.8. What can you conclude about ‖∂h_t/∂h_k‖?
Check yourself
An RNN layer is nn.RNN(input_size=65, hidden_size=128). How many parameters does it have?
Remember
- ✓Elman cell: ;
nn.RNNhas parameters. - ✓BPTT: ; the gradient across a lag is a product of factors.
- ✓Pascanu et al. (2013): is sufficient for vanishing, is necessary (not sufficient) for exploding.
- ✓Clip the gradient norm against explosions; use orthogonal init, gating (LSTM/GRU) or other architectures against vanishing.
- ✓
h_0andh_nare(layers·dirs, N, H)even withbatch_first=True.