Vanilla RNN

The Elman cell h_t = tanh(W_xh x_t + W_hh h_{t−1} + b): one affine map and a tanh per step. Its BPTT gradient is a product of diag(1 − h²)·W_hh factors, so it vanishes when W_hh's largest singular value is below 1 and can explode only above 1.

14 min read1000 wordscore0 viewsupdated 2026-10-12

Definition

Vanilla RNN (also Elman network or simple RNN) is the recurrent cell ht=tanh⁡(Wxhxt+Whhht−1+b)h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b): the new hidden state is a squashed affine function of the current input and the previous state. An output layer yt=Whyht+byy_t=W_{hy}h_t+b_y reads the state. There are no gates; the whole memory mechanism is the matrix WhhW_{hh}.

This is the cell that Recurrent neural network (RNN) describes in general terms, made concrete. Elman (1990) introduced it as a feed-forward net whose hidden layer is copied into "context units" and fed back at the next step. It is the baseline every gated cell is compared against, and it is still worth understanding in detail, because its failure mode, the product of Jacobians across time, is exactly what LSTM, GRU, orthogonal initialisation and gradient clipping were designed to fix.

Two papers frame its training. Bengio, Simard and Frasconi (1994) showed that learning long-range dependencies with gradient descent is hard in general. Pascanu, Mikolov and Bengio (2013), On the difficulty of training recurrent neural networks (ICML), made the analysis precise for this cell: they bounded the BPTT Jacobians by the largest singular value of WhhW_{hh}, explained exploding gradients as "cliffs" in the loss surface, and proposed rescaling the gradient norm (clipping) as the remedy. Karpathy's min-char-rnn (2015), about a hundred lines of NumPy implementing exactly this cell on characters, made it the standard teaching example.

The cell

at=Wxhxt+Whhht−1+b,ht=tanh⁡(at),y^t=softmax⁡(Whyht+by)a_t=W_{xh}x_t+W_{hh}h_{t-1}+b,\qquad h_t=\tanh(a_t),\qquad \hat y_t=\operatorname{softmax}(W_{hy}h_t+b_y)
xt∈RIx_t\in\mathbb R^{I}
input at step tt (a one-hot character or an embedding)
ht∈RHh_t\in\mathbb R^{H}
hidden state, h0=0h_0=0 unless given
Wxh∈RH×IW_{xh}\in\mathbb R^{H\times I}
input-to-hidden weights (PyTorch: weight_ih_l0)
Whh∈RH×HW_{hh}\in\mathbb R^{H\times H}
hidden-to-hidden (recurrent) weights (weight_hh_l0)
bb
bias; PyTorch keeps two, bih+bhhb_{ih}+b_{hh}, for cuDNN compatibility
Why, byW_{hy},\,b_y
readout to VV classes, usually a separate nn.Linear

Counting parameters: WxhW_{xh} has HIHI entries, WhhW_{hh} has H2H^2, and the two bias vectors 2H2H, so nn.RNN(I, H) has H(I+H+2)H(I+H+2) parameters. A character model with I=65I=65 symbols (tiny Shakespeare) and H=128H=128 has 128⋅195=24,960128\cdot195=24{,}960 in the cell plus 128⋅65+65=8,385128\cdot65+65=8{,}385 in the readout. The recurrent matrix dominates once H>IH>I. Compute is two matrix–vector products per step, O(H(I+H))O(H(I+H)), and the steps must run one after another.

At a glance

Update
ht=tanh⁡(Wxhxt+Whhht−1+b)h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b)
Parameters
H(I+H+2)H(I+H+2) in PyTorch (two biases); H(I+H+1)H(I+H+1) with one
Local Jacobian
∂ht/∂ht−1=diag⁡(1−ht2) Whh\partial h_t/\partial h_{t-1}=\operatorname{diag}(1-h_t^2)\,W_{hh}
Vanishing (sufficient)
σmax⁡(Whh)<1\sigma_{\max}(W_{hh})<1 with tanh (Pascanu et al., 2013)
Exploding (necessary)
σmax⁡(Whh)>1\sigma_{\max}(W_{hh})>1
Practical reach
dependencies beyond a few tens of steps are hard to learn
PyTorch
nn.RNN, nn.RNNCell, nonlinearity="tanh" or "relu"

The full BPTT gradient

Let δt=∂L/∂at\delta_t=\partial L/\partial a_t be the error at the pre-activation of step tt. Backpropagation through time runs the recurrence backwards. Each step receives the loss's direct gradient through the readout and the gradient coming back from step t+1t+1:

δt=(1−ht2)⊙(Why⊤(y^t−yt)+Whh⊤δt+1),∂L∂Whh=∑tδt ht−1⊤,∂L∂Wxh=∑tδt xt⊤\delta_t=(1-h_t^2)\odot\Big(W_{hy}^{\top}(\hat y_t-y_t)+W_{hh}^{\top}\delta_{t+1}\Big),\qquad \frac{\partial L}{\partial W_{hh}}=\sum_{t}\delta_t\,h_{t-1}^{\top},\quad \frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t\,x_t^{\top}
y^t−yt\hat y_t-y_t
softmax-plus-cross-entropy gradient at step t (one-hot target)
Whh⊤δt+1W_{hh}^{\top}\delta_{t+1}
the error flowing back in time from the next step
1−ht21-h_t^2
tanh⁡′(at)\tanh'(a_t): near 0 when a unit is saturated at ±1\pm1
δT+1=0\delta_{T+1}=0
nothing comes back from beyond the last step

Unrolling the recursion, the error that the loss at step tt sends back to step kk passes through a product of local Jacobians:

∂ht∂hk=∏j=k+1tdiag⁡ ⁣(1−hj2) Whh,∥∂ht∂hk∥2≤(γ σmax⁡(Whh))t−k\frac{\partial h_t}{\partial h_k}=\prod_{j=k+1}^{t}\operatorname{diag}\!\big(1-h_j^{2}\big)\,W_{hh},\qquad \Big\|\frac{\partial h_t}{\partial h_k}\Big\|_2\le\big(\gamma\,\sigma_{\max}(W_{hh})\big)^{t-k}
σmax⁡(Whh)\sigma_{\max}(W_{hh})
largest singular value of the recurrent matrix (for a normal matrix, equal to the spectral radius ρ\rho)
γ\gamma
bound on ∣φ′∣|\varphi'|: 1 for tanh, 1/4 for the logistic sigmoid
t−kt-k
the time lag: the bound is exponential in it

Pascanu et al. (2013) read two conditions off this bound. If σmax⁡(Whh)<1/γ\sigma_{\max}(W_{hh})<1/\gamma, every factor is a contraction and long-term contributions vanish exponentially; this is sufficient. If gradients explode, then σmax⁡(Whh)>1/γ\sigma_{\max}(W_{hh})>1/\gamma must hold; this is only necessary. The asymmetry matters. A recurrent matrix with large singular values usually drives the units into saturation, where 1−hj2≈01-h_j^2\approx0, and the diagonal factors then kill the product. Exploding happens in a narrower regime: WhhW_{hh} expanding, but the states still in tanh's linear range. For a linear RNN the picture is exact: ∂ht/∂hk=Whh t−k\partial h_t/\partial h_k=W_{hh}^{\,t-k}, which grows or shrinks like ρt−k\rho^{t-k} along the leading eigenvector.

hₜ = tanh(W_xh xₜ + W_hh hₜ₋₁), W_hh = ρ·Q (Q orthogonal, all singular values = ρ)hₜt = 1t = 60 · mean |hₜ| = 0.411 (1 = fully saturated)1e61e311e−31e−61e−91e−121e−1501020304050lag k = T − t‖∂h₆₀/∂h₆₀₋ₖ‖₂ (log scale)lag 50: tanh RNN 1.23e−7 · linear RNN ρ^50 = 5.15e−3 · vanishedper-step factor over 50 lags 0.727 · peak norm 1.00 · ━ tanh ┄ ρᵏ
fig.An 8-unit tanh RNN driven by 60 random 4-d inputs, with W_hh = ρ·Q for a fixed orthogonal Q, so every singular value equals ρ. Top: the hidden states hₜ (blue positive, orange negative). Bottom: ‖∂h₆₀/∂h₆₀₋ₖ‖₂ computed by explicit BPTT (violet) against the linear-RNN prediction ρᵏ (dashed). At the default ρ = 0.9, input scale 0.5, the norm at lag 50 is 1.2e−7, far below ρ⁵⁰ = 5.2e−3, because tanh slopes below 1 shrink every factor further. Set the input scale to 0: the states stay at 0 and the two curves coincide exactly (117 at lag 50 for ρ = 1.1). With ρ = 1.3 and input scale 0.1 the gradient grows to 11 by lag 59 while the linear RNN would give 5e5 at lag 50. Push ρ to 2.0: mean |hₜ| rises to 0.77, the units saturate and the norm at lag 50 collapses to 1.4e−12 even though ρ > 1.

Taming the gradient

Exploding gradients are rare but catastrophic: one step on a cliff throws the weights far away and the loss spikes or turns into NaN. The standard fix is gradient norm clipping (Pascanu et al., 2013): compute the norm of the full gradient vector and, if it exceeds a threshold cc, rescale it to length cc while keeping its direction. Mikolov had earlier clipped element-wise in his RNN language models. See Gradient clipping for the full story.

g←{g∥g∥≤cc∥g∥ g∥g∥>cg\leftarrow\begin{cases}g & \|g\|\le c\\[2pt] \dfrac{c}{\|g\|}\,g & \|g\|>c\end{cases}
gg
the concatenated gradient of all parameters
cc
threshold, e.g. 1–5 for RNN language models; set it slightly above the typical norm

Vanishing gradients have no such one-line fix, because a tiny gradient is not wrong, just uninformative. The remedies change the model. Initialise WhhW_{hh} orthogonal (all singular values 1; Saxe et al., 2014) or as the identity with ReLU units (IRNN; Le, Jaitly and Hinton, 2015). Constrain WhhW_{hh} to be unitary (Arjovsky et al., 2016). Add a soft penalty that keeps ∥∂ht/∂ht−1∥\|\partial h_{t}/\partial h_{t-1}\| near 1 (Pascanu et al.'s regulariser). Or, most commonly, replace the cell with an additive, gated one: Long short-term memory (LSTM) or Gated recurrent unit (GRU).

Worked example: a 2-unit character model on "hell" → "ello"

Vocabulary {h,e,l,o}\{h,e,l,o\}, one-hot inputs, H=2H=2, all biases 0. The columns of WxhW_{xh} are the characters' input vectors h ⁣:(1,0)h\!:(1,0), e ⁣:(0,1)e\!:(0,1), l ⁣:(−1,0.5)l\!:(-1,0.5), o ⁣:(0.5,−1)o\!:(0.5,-1). The recurrent matrix is Whh=(0.5−0.40.30.6)W_{hh}=\begin{pmatrix}0.5&-0.4\\0.3&0.6\end{pmatrix} (singular values 0.723 and 0.581, spectral radius 0.648), and the readout rows are h ⁣:(1,−1)h\!:(1,-1), e ⁣:(−1,1.5)e\!:(-1,1.5), l ⁣:(0.5,1)l\!:(0.5,1), o ⁣:(−0.5,−0.5)o\!:(-0.5,-0.5). The model reads "hell" and should predict the next character at each step: "ello".

tinput → targeta_th_tp(target)−log pargmax
1h → e(1.0000, 0.0000)(0.7616, 0.0000)0.09822.3209h
2e → l(0.3808, 1.2285)(0.3634, 0.8421)0.43430.8340l
3l → l(−1.1552, 1.1143)(−0.8195, 0.8056)0.14441.9350e
4l → o(−1.7320, 0.7375)(−0.9393, 0.6276)0.12832.0532e
Forward pass of the untrained model (verified in Node)

Reading the example

  1. 1Step 2: a2=Wxhe+Whhh1=(0,1)+(0.5⋅0.7616, 0.3⋅0.7616)=(0.3808, 1.2285)a_2=W_{xh}e+W_{hh}h_1=(0,1)+(0.5\cdot0.7616,\ 0.3\cdot0.7616)=(0.3808,\,1.2285), so h2=tanh⁡a2=(0.3634, 0.8421)h_2=\tanh a_2=(0.3634,\,0.8421).
  2. 2Logits Whyh2=(−0.4787, 0.8998, 1.0238, −0.6028)W_{hy}h_2=(-0.4787,\,0.8998,\,1.0238,\,-0.6028); the softmax gives p(l)=0.4343p(l)=0.4343, the one correct prediction so far.
  3. 3Steps 3 and 4 both read "l" but need different answers (l, then o). Only the hidden state can tell them apart: h3=(−0.82,0.81)h_3=(-0.82,0.81) and h4=(−0.94,0.63)h_4=(-0.94,0.63) differ because h2≠h3h_2\ne h_3. Training must shape WhhW_{hh} so that this difference is enough.
  4. 4Mean loss 7.1431/4=1.7867.1431/4=1.786 nats, worse than the uniform guess ln⁡4=1.386\ln4=1.386: the weights are untrained.
  5. 5Backward in time: ∂h2/∂h1=diag⁡(1−h22)Whh=(0.4340−0.34720.08720.1745)\partial h_2/\partial h_1=\operatorname{diag}(1-h_2^2)W_{hh}=\begin{pmatrix}0.4340&-0.3472\\0.0872&0.1745\end{pmatrix}, and over three steps ∥∂h4/∂h1∥2=0.0372\|\partial h_4/\partial h_1\|_2=0.0372, already a 27-fold shrink with σmax⁡=0.72\sigma_{\max}=0.72.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3# --- shapes ---------------------------------------------------------------
4rnn = nn.RNN(input_size=4, hidden_size=2, nonlinearity="tanh", batch_first=True)
5x = torch.randn(3, 5, 4) # (N, T, I) because batch_first=True
6h0 = torch.zeros(1, 3, 2) # (num_layers * num_directions, N, H) -- NOT batch-first
7out, h_n = rnn(x, h0)
8print(out.shape, h_n.shape) # torch.Size([3, 5, 2]) torch.Size([1, 3, 2])
9assert torch.allclose(out[:, -1], h_n[0])
10print(sum(p.numel() for p in rnn.parameters())) # 2*(4+2+2) = 16
11
12# --- the same cell by hand ---------------------------------------------------
13W_ih, W_hh, b_ih, b_hh = rnn.weight_ih_l0, rnn.weight_hh_l0, rnn.bias_ih_l0, rnn.bias_hh_l0
14h = h0[0]
15for t in range(x.shape[1]):
16 h = torch.tanh(x[:, t] @ W_ih.T + b_ih + h @ W_hh.T + b_hh)
17assert torch.allclose(h, h_n[0], atol=1e-6)
18
19# --- char-level language model ------------------------------------------------
20text = "hello world, hello rnn. " * 50
21vocab = sorted(set(text)); stoi = {c: i for i, c in enumerate(vocab)}
22data = torch.tensor([stoi[c] for c in text])
23V, H, T = len(vocab), 128, 32
24
25class CharRNN(nn.Module):
26 def __init__(self):
27 super().__init__()
28 self.rnn = nn.RNN(V, H, batch_first=True)
29 self.head = nn.Linear(H, V)
30 def forward(self, idx, h=None):
31 out, h = self.rnn(F.one_hot(idx, V).float(), h)
32 return self.head(out), h
33
34model = CharRNN()
35nn.init.orthogonal_(model.rnn.weight_hh_l0) # all singular values 1 at the start
36opt = torch.optim.Adam(model.parameters(), lr=3e-3)
37for step in range(300):
38 i = torch.randint(0, len(data) - T - 1, (16,))
39 xb = torch.stack([data[j:j + T] for j in i]); yb = torch.stack([data[j + 1:j + T + 1] for j in i])
40 logits, _ = model(xb)
41 loss = F.cross_entropy(logits.reshape(-1, V), yb.reshape(-1))
42 opt.zero_grad(); loss.backward()
43 norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # returns the pre-clip norm
44 opt.step()
nn.RNN shapes, the cell by hand, and a char-level training step with clipping

Use when

  • +Minimal: one affine map and a tanh, H(I+H+2)H(I+H+2) parameters
  • +Cheapest recurrent cell per step; easy to analyse exactly
  • +Fine for short dependencies (tens of steps) and as a baseline
  • +Saturating tanh keeps states bounded in [−1,1][-1,1]

Avoid when

  • −Long-range gradients vanish whenever σmax⁡(Whh)<1\sigma_{\max}(W_{hh})<1, and saturation makes them vanish even above 1
  • −Exploding gradients on loss cliffs: clipping is mandatory
  • −Sensitive to the initialisation of the recurrent matrix
  • −Outperformed by LSTM and GRU on almost every task with long-range structure

Check yourself

The largest singular value of a tanh RNN's W_hh is 0.8. What can you conclude about ‖∂h_t/∂h_k‖?

Check yourself

An RNN layer is nn.RNN(input_size=65, hidden_size=128). How many parameters does it have?

Remember

  • ✓Elman cell: ht=tanh⁡(Wxhxt+Whhht−1+b)h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b); nn.RNN has H(I+H+2)H(I+H+2) parameters.
  • ✓BPTT: δt=(1−ht2)⊙(Why⊤et+Whh⊤δt+1)\delta_t=(1-h_t^2)\odot(W_{hy}^\top e_t+W_{hh}^\top\delta_{t+1}); the gradient across a lag is a product of diag⁡(1−h2)Whh\operatorname{diag}(1-h^2)W_{hh} factors.
  • ✓Pascanu et al. (2013): σmax⁡(Whh)<1\sigma_{\max}(W_{hh})<1 is sufficient for vanishing, >1>1 is necessary (not sufficient) for exploding.
  • ✓Clip the gradient norm against explosions; use orthogonal init, gating (LSTM/GRU) or other architectures against vanishing.
  • ✓h_0 and h_n are (layers·dirs, N, H) even with batch_first=True.
← back to Deep Learning roadmap