# Vanilla RNN

> The Elman cell h_t = tanh(W_xh x_t + W_hh h_{t−1} + b): one affine map and a tanh per step. Its BPTT gradient is a product of diag(1 − h²)·W_hh factors, so it vanishes when W_hh's largest singular value is below 1 and can explode only above 1.

Source: https://www.mlfoundry.in/learn/vanilla-rnn  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 14 min read · updated 2026-10-12

**Vanilla RNN:** (also **Elman network** or **simple RNN**) is the recurrent cell $h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b)$: the new hidden state is a squashed affine function of the current input and the previous state. An output layer $y_t=W_{hy}h_t+b_y$ reads the state. There are no gates; the whole memory mechanism is the matrix $W_{hh}$.

This is the cell that **Recurrent neural network (RNN)** describes in general terms, made concrete. Elman (1990) introduced it as a feed-forward net whose hidden layer is copied into "context units" and fed back at the next step. It is the baseline every gated cell is compared against, and it is still worth understanding in detail, because its failure mode, the product of Jacobians across time, is exactly what LSTM, GRU, orthogonal initialisation and gradient clipping were designed to fix.

Two papers frame its training. **Bengio, Simard and Frasconi (1994)** showed that learning long-range dependencies with gradient descent is hard in general. **Pascanu, Mikolov and Bengio (2013)**, *On the difficulty of training recurrent neural networks* (ICML), made the analysis precise for this cell: they bounded the BPTT Jacobians by the largest singular value of $W_{hh}$, explained exploding gradients as "cliffs" in the loss surface, and proposed rescaling the gradient norm (clipping) as the remedy. Karpathy's *min-char-rnn* (2015), about a hundred lines of NumPy implementing exactly this cell on characters, made it the standard teaching example.

## The cell

$$
a_t=W_{xh}x_t+W_{hh}h_{t-1}+b,\qquad h_t=\tanh(a_t),\qquad \hat y_t=\operatorname{softmax}(W_{hy}h_t+b_y)
$$

- $x_t\in\mathbb R^{I}$: input at step $t$ (a one-hot character or an embedding)
- $h_t\in\mathbb R^{H}$: hidden state, $h_0=0$ unless given
- $W_{xh}\in\mathbb R^{H\times I}$: input-to-hidden weights (PyTorch: weight_ih_l0)
- $W_{hh}\in\mathbb R^{H\times H}$: hidden-to-hidden (recurrent) weights (weight_hh_l0)
- $b$: bias; PyTorch keeps two, $b_{ih}+b_{hh}$, for cuDNN compatibility
- $W_{hy},\,b_y$: readout to $V$ classes, usually a separate `nn.Linear`

Counting parameters: $W_{xh}$ has $HI$ entries, $W_{hh}$ has $H^2$, and the two bias vectors $2H$, so `nn.RNN(I, H)` has $H(I+H+2)$ parameters. A character model with $I=65$ symbols (tiny Shakespeare) and $H=128$ has $128\cdot195=24{,}960$ in the cell plus $128\cdot65+65=8{,}385$ in the readout. The recurrent matrix dominates once $H>I$. Compute is two matrix–vector products per step, $O(H(I+H))$, and the steps must run one after another.

**At a glance**

- **Update:** $h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b)$
- **Parameters:** $H(I+H+2)$ in PyTorch (two biases); $H(I+H+1)$ with one
- **Local Jacobian:** $\partial h_t/\partial h_{t-1}=\operatorname{diag}(1-h_t^2)\,W_{hh}$
- **Vanishing (sufficient):** $\sigma_{\max}(W_{hh})<1$ with tanh (Pascanu et al., 2013)
- **Exploding (necessary):** $\sigma_{\max}(W_{hh})>1$
- **Practical reach:** dependencies beyond a few tens of steps are hard to learn
- **PyTorch:** `nn.RNN`, `nn.RNNCell`, `nonlinearity="tanh"` or `"relu"`

## The full BPTT gradient

Let $\delta_t=\partial L/\partial a_t$ be the error at the pre-activation of step $t$. Backpropagation through time runs the recurrence backwards. Each step receives the loss's direct gradient through the readout and the gradient coming back from step $t+1$:

$$
\delta_t=(1-h_t^2)\odot\Big(W_{hy}^{\top}(\hat y_t-y_t)+W_{hh}^{\top}\delta_{t+1}\Big),\qquad \frac{\partial L}{\partial W_{hh}}=\sum_{t}\delta_t\,h_{t-1}^{\top},\quad \frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t\,x_t^{\top}
$$

- $\hat y_t-y_t$: softmax-plus-cross-entropy gradient at step t (one-hot target)
- $W_{hh}^{\top}\delta_{t+1}$: the error flowing back in time from the next step
- $1-h_t^2$: $\tanh'(a_t)$: near 0 when a unit is saturated at $\pm1$
- $\delta_{T+1}=0$: nothing comes back from beyond the last step

Unrolling the recursion, the error that the loss at step $t$ sends back to step $k$ passes through a product of local Jacobians:

$$
\frac{\partial h_t}{\partial h_k}=\prod_{j=k+1}^{t}\operatorname{diag}\!\big(1-h_j^{2}\big)\,W_{hh},\qquad \Big\|\frac{\partial h_t}{\partial h_k}\Big\|_2\le\big(\gamma\,\sigma_{\max}(W_{hh})\big)^{t-k}
$$

- $\sigma_{\max}(W_{hh})$: largest singular value of the recurrent matrix (for a normal matrix, equal to the spectral radius $\rho$)
- $\gamma$: bound on $|\varphi'|$: 1 for tanh, 1/4 for the logistic sigmoid
- $t-k$: the time lag: the bound is exponential in it

Pascanu et al. (2013) read two conditions off this bound. If $\sigma_{\max}(W_{hh})<1/\gamma$, every factor is a contraction and long-term contributions **vanish** exponentially; this is **sufficient**. If gradients **explode**, then $\sigma_{\max}(W_{hh})>1/\gamma$ must hold; this is only **necessary**. The asymmetry matters. A recurrent matrix with large singular values usually drives the units into saturation, where $1-h_j^2\approx0$, and the diagonal factors then kill the product. Exploding happens in a narrower regime: $W_{hh}$ expanding, but the states still in tanh's linear range. For a **linear** RNN the picture is exact: $\partial h_t/\partial h_k=W_{hh}^{\,t-k}$, which grows or shrinks like $\rho^{t-k}$ along the leading eigenvector.

> Figure: An 8-unit tanh RNN driven by 60 random 4-d inputs, with W_hh = ρ·Q for a fixed orthogonal Q, so every singular value equals ρ. Top: the hidden states hₜ (blue positive, orange negative). Bottom: ‖∂h₆₀/∂h₆₀₋ₖ‖₂ computed by explicit BPTT (violet) against the linear-RNN prediction ρᵏ (dashed). At the default ρ = 0.9, input scale 0.5, the norm at lag 50 is 1.2e−7, far below ρ⁵⁰ = 5.2e−3, because tanh slopes below 1 shrink every factor further. Set the input scale to 0: the states stay at 0 and the two curves coincide exactly (117 at lag 50 for ρ = 1.1). With ρ = 1.3 and input scale 0.1 the gradient grows to 11 by lag 59 while the linear RNN would give 5e5 at lag 50. Push ρ to 2.0: mean |hₜ| rises to 0.77, the units saturate and the norm at lag 50 collapses to 1.4e−12 even though ρ > 1.

## Taming the gradient

**Exploding** gradients are rare but catastrophic: one step on a cliff throws the weights far away and the loss spikes or turns into NaN. The standard fix is **gradient norm clipping** (Pascanu et al., 2013): compute the norm of the full gradient vector and, if it exceeds a threshold $c$, rescale it to length $c$ while keeping its direction. Mikolov had earlier clipped element-wise in his RNN language models. See **Gradient clipping** for the full story.

$$
g\leftarrow\begin{cases}g & \|g\|\le c\\[2pt] \dfrac{c}{\|g\|}\,g & \|g\|>c\end{cases}
$$

- $g$: the concatenated gradient of all parameters
- $c$: threshold, e.g. 1–5 for RNN language models; set it slightly above the typical norm

**Vanishing** gradients have no such one-line fix, because a tiny gradient is not wrong, just uninformative. The remedies change the model. Initialise $W_{hh}$ orthogonal (all singular values 1; Saxe et al., 2014) or as the identity with ReLU units (IRNN; Le, Jaitly and Hinton, 2015). Constrain $W_{hh}$ to be unitary (Arjovsky et al., 2016). Add a soft penalty that keeps $\|\partial h_{t}/\partial h_{t-1}\|$ near 1 (Pascanu et al.'s regulariser). Or, most commonly, replace the cell with an additive, gated one: **Long short-term memory (LSTM)** or **Gated recurrent unit (GRU)**.

> **Why initialisation is everything here:** A deep feed-forward net has different weights in every layer, so their effects partly average out. The vanilla RNN multiplies by the **same** $W_{hh}$ at every step, so whatever $W_{hh}$ does to one direction, it does $T$ times. That is why its singular values, not just its variance, decide whether training works, and why orthogonal initialisation is so effective for this cell.

## Worked example: a 2-unit character model on "hell" → "ello"

Vocabulary $\{h,e,l,o\}$, one-hot inputs, $H=2$, all biases 0. The columns of $W_{xh}$ are the characters' input vectors $h\!:(1,0)$, $e\!:(0,1)$, $l\!:(-1,0.5)$, $o\!:(0.5,-1)$. The recurrent matrix is $W_{hh}=\begin{pmatrix}0.5&-0.4\\0.3&0.6\end{pmatrix}$ (singular values 0.723 and 0.581, spectral radius 0.648), and the readout rows are $h\!:(1,-1)$, $e\!:(-1,1.5)$, $l\!:(0.5,1)$, $o\!:(-0.5,-0.5)$. The model reads "hell" and should predict the next character at each step: "ello".

Forward pass of the untrained model (verified in Node)

| t | input → target | a_t | h_t | p(target) | −log p | argmax |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | h → e | (1.0000, 0.0000) | (0.7616, 0.0000) | 0.0982 | 2.3209 | h |
| 2 | e → l | (0.3808, 1.2285) | (0.3634, 0.8421) | 0.4343 | 0.8340 | l |
| 3 | l → l | (−1.1552, 1.1143) | (−0.8195, 0.8056) | 0.1444 | 1.9350 | e |
| 4 | l → o | (−1.7320, 0.7375) | (−0.9393, 0.6276) | 0.1283 | 2.0532 | e |

**Reading the example**

1. Step 2: $a_2=W_{xh}e+W_{hh}h_1=(0,1)+(0.5\cdot0.7616,\ 0.3\cdot0.7616)=(0.3808,\,1.2285)$, so $h_2=\tanh a_2=(0.3634,\,0.8421)$.
2. Logits $W_{hy}h_2=(-0.4787,\,0.8998,\,1.0238,\,-0.6028)$; the softmax gives $p(l)=0.4343$, the one correct prediction so far.
3. Steps 3 and 4 both read "l" but need different answers (l, then o). Only the hidden state can tell them apart: $h_3=(-0.82,0.81)$ and $h_4=(-0.94,0.63)$ differ because $h_2\ne h_3$. Training must shape $W_{hh}$ so that this difference is enough.
4. Mean loss $7.1431/4=1.786$ nats, worse than the uniform guess $\ln4=1.386$: the weights are untrained.
5. Backward in time: $\partial h_2/\partial h_1=\operatorname{diag}(1-h_2^2)W_{hh}=\begin{pmatrix}0.4340&-0.3472\\0.0872&0.1745\end{pmatrix}$, and over three steps $\|\partial h_4/\partial h_1\|_2=0.0372$, already a 27-fold shrink with $\sigma_{\max}=0.72$.

```python
import torch, torch.nn as nn, torch.nn.functional as F

# --- shapes ---------------------------------------------------------------
rnn = nn.RNN(input_size=4, hidden_size=2, nonlinearity="tanh", batch_first=True)
x = torch.randn(3, 5, 4)                 # (N, T, I) because batch_first=True
h0 = torch.zeros(1, 3, 2)                # (num_layers * num_directions, N, H) -- NOT batch-first
out, h_n = rnn(x, h0)
print(out.shape, h_n.shape)              # torch.Size([3, 5, 2]) torch.Size([1, 3, 2])
assert torch.allclose(out[:, -1], h_n[0])
print(sum(p.numel() for p in rnn.parameters()))   # 2*(4+2+2) = 16

# --- the same cell by hand ---------------------------------------------------
W_ih, W_hh, b_ih, b_hh = rnn.weight_ih_l0, rnn.weight_hh_l0, rnn.bias_ih_l0, rnn.bias_hh_l0
h = h0[0]
for t in range(x.shape[1]):
    h = torch.tanh(x[:, t] @ W_ih.T + b_ih + h @ W_hh.T + b_hh)
assert torch.allclose(h, h_n[0], atol=1e-6)

# --- char-level language model ------------------------------------------------
text = "hello world, hello rnn. " * 50
vocab = sorted(set(text)); stoi = {c: i for i, c in enumerate(vocab)}
data = torch.tensor([stoi[c] for c in text])
V, H, T = len(vocab), 128, 32

class CharRNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.rnn = nn.RNN(V, H, batch_first=True)
        self.head = nn.Linear(H, V)
    def forward(self, idx, h=None):
        out, h = self.rnn(F.one_hot(idx, V).float(), h)
        return self.head(out), h

model = CharRNN()
nn.init.orthogonal_(model.rnn.weight_hh_l0)   # all singular values 1 at the start
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
for step in range(300):
    i = torch.randint(0, len(data) - T - 1, (16,))
    xb = torch.stack([data[j:j + T] for j in i]); yb = torch.stack([data[j + 1:j + T + 1] for j in i])
    logits, _ = model(xb)
    loss = F.cross_entropy(logits.reshape(-1, V), yb.reshape(-1))
    opt.zero_grad(); loss.backward()
    norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # returns the pre-clip norm
    opt.step()
```

nn.RNN shapes, the cell by hand, and a char-level training step with clipping

> **h_0 ignores batch_first:** `batch_first=True` changes only the layout of the input and of `out`. The initial and final hidden states are always `(num_layers * num_directions, N, H)`. Passing a `(N, 1, H)` tensor as `h0` either errors or, when $N=1$, silently does the wrong thing in user code that reshapes it.

> **ReLU instead of tanh:** `nonlinearity="relu"` removes saturation, so nothing caps the product of $W_{hh}$ factors: with $\sigma_{\max}>1$ activations and gradients can blow up quickly. It works when paired with identity or carefully scaled initialisation (IRNN) and clipping; otherwise keep tanh.

> **Default initialisation:** PyTorch initialises every RNN weight and bias from $\mathcal U(-1/\sqrt H,\,1/\sqrt H)$. For long sequences it often pays to override `weight_hh_l0` with `nn.init.orthogonal_` (as the code above does) and to zero the biases.

**Strengths**

- Minimal: one affine map and a tanh, $H(I+H+2)$ parameters
- Cheapest recurrent cell per step; easy to analyse exactly
- Fine for short dependencies (tens of steps) and as a baseline
- Saturating tanh keeps states bounded in $[-1,1]$

**Limitations**

- Long-range gradients vanish whenever $\sigma_{\max}(W_{hh})<1$, and saturation makes them vanish even above 1
- Exploding gradients on loss cliffs: clipping is mandatory
- Sensitive to the initialisation of the recurrent matrix
- Outperformed by LSTM and GRU on almost every task with long-range structure

**Question:** The largest singular value of a tanh RNN's W_hh is 0.8. What can you conclude about ‖∂h_t/∂h_k‖?

- A. It is at most 0.8^(t−k): long-range gradients are guaranteed to vanish
- B. It grows like 0.8^(−(t−k))
- C. Nothing, because tanh can amplify gradients
- D. It is exactly 0.8^(t−k)

**Answer:** A. It is at most 0.8^(t−k): long-range gradients are guaranteed to vanish — Each factor $\operatorname{diag}(1-h_j^2)W_{hh}$ has norm at most $1\cdot0.8$ because $|\tanh'|\le1$. The product is bounded by $0.8^{t-k}$; saturation can only make it smaller. Equality would need a linear RNN with a normal $W_{hh}$.

**Question:** An RNN layer is nn.RNN(input_size=65, hidden_size=128). How many parameters does it have?

- A. 24,960
- B. 24,832
- C. 16,384
- D. 33,345

**Answer:** A. 24,960 — $H(I+H+2)=128\cdot(65+128+2)=24{,}960$: $W_{ih}$ is $128\times65$, $W_{hh}$ is $128\times128$, plus two bias vectors of 128. 24,832 forgets one bias; 33,345 adds the `nn.Linear(128, 65)` readout.

**Key points**

- Elman cell: $h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b)$; `nn.RNN` has $H(I+H+2)$ parameters.
- BPTT: $\delta_t=(1-h_t^2)\odot(W_{hy}^\top e_t+W_{hh}^\top\delta_{t+1})$; the gradient across a lag is a product of $\operatorname{diag}(1-h^2)W_{hh}$ factors.
- Pascanu et al. (2013): $\sigma_{\max}(W_{hh})<1$ is sufficient for vanishing, $>1$ is necessary (not sufficient) for exploding.
- Clip the gradient norm against explosions; use orthogonal init, gating (LSTM/GRU) or other architectures against vanishing.
- `h_0` and `h_n` are `(layers·dirs, N, H)` even with `batch_first=True`.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
