# Recurrent neural network (RNN)

> A network that reads a sequence one step at a time and carries a hidden state, h_t = f(h_{t−1}, x_t), with the same weights at every step. Unrolled in time it is a deep net trained by backpropagation through time, whose gradients shrink or blow up with the time lag.

Source: https://www.mlfoundry.in/learn/recurrent-neural-network-rnn  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 12 min read · updated 2026-10-12

**Recurrent neural network:** (RNN) is a neural network with a loop. At every time step $t$ it combines the current input $x_t$ with its own previous **hidden state** $h_{t-1}$ to produce a new state $h_t=f_\theta(h_{t-1},x_t)$, and optionally an output $y_t=g_\theta(h_t)$. The same parameters $\theta$ are used at every step, so one small cell can process a sequence of any length.

A feed-forward network maps one fixed-size input to one output. Text, speech, sensor streams and actions arrive as **sequences** whose length varies and whose meaning depends on order: "dog bites man" is not "man bites dog". You could concatenate a fixed window of inputs, but then the window caps how far back the model can look, and a pattern learned at position 3 has to be relearned at position 30. A recurrent network fixes both problems with one idea: keep a running summary $h_t$ of everything seen so far and update it with the **same** function at each step.

The idea grew out of the connectionist work of the 1980s. Hopfield (1982) used recurrent connections for associative memory (see **Hopfield network**), but those nets settle to fixed points rather than process sequences. **Jordan (1986)** fed the network's *outputs* back as extra "state" inputs to produce serial order. **Elman (1990)**, *Finding structure in time*, fed back the *hidden layer* through "context units"; trained on next-word prediction, his network discovered word classes such as nouns and verbs from order alone. That hidden-to-hidden loop is the cell now called the **simple** or **vanilla RNN**. Rumelhart, Hinton and Williams (1986) had already shown how to train such nets by unfolding them in time, and Werbos (1990) gave the method its standard name, **backpropagation through time** (BPTT). The trouble came next. Hochreiter's 1991 diploma thesis and **Bengio, Simard and Frasconi (1994)**, *Learning long-term dependencies with gradient descent is difficult*, showed that the gradient signal decays (or explodes) exponentially with the time lag, which motivated the LSTM (1997) and every gated design since.

## The recurrence

$$
h_t=f_\theta\big(h_{t-1},\,x_t\big),\qquad y_t=g_\theta(h_t),\qquad h_0=\text{given (usually }0)
$$

- $x_t$: input at step $t$ (a word embedding, an audio frame, a sensor reading)
- $h_t$: hidden state: a fixed-size vector that summarises $x_1,\dots,x_t$
- $f_\theta$: the cell. Vanilla RNN: $\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b)$; LSTM and GRU add gates
- $g_\theta$: readout, e.g. logits $W_{hy}h_t+b_y$ followed by a softmax
- $\theta$: shared over time: the same weights at $t=1$ and $t=1000$

Unrolling the loop for $T$ steps turns the RNN into a $T$-layer feed-forward network in which every layer has the same weights: $h_T=f(f(\cdots f(h_0,x_1)\cdots,x_{T-1}),x_T)$. Two consequences follow. The number of parameters does not depend on $T$, so the model generalises across positions and lengths. And training it means backpropagating through a network as deep as the sequence is long.

**At a glance**

- **Input:** a sequence $x_1,\dots,x_T$ of vectors, any length $T$
- **State:** a fixed-size $h_t\in\mathbb R^H$ carried from step to step
- **Parameters:** independent of $T$ (weight sharing over time)
- **Compute:** $O(T)$ cell evaluations, inherently **sequential** in $t$
- **Memory at inference:** $O(H)$: only the current state is kept, however long the stream
- **Training:** backpropagation through time; memory $O(T\cdot H)$ for the stored states
- **Main weakness:** gradients across long lags vanish or explode, so plain RNNs struggle beyond tens of steps

> Figure: A one-unit RNN, hₜ = tanh(w·hₜ₋₁ + 1.5·xₜ − 0.3), folded (left) and unrolled over 8 steps on the input bits 1 0 0 1 0 0 0 1. Each box shows hₜ, and the same w sits on every arrow. Click a step to see its update written out. The topology buttons change where inputs are fed and outputs are read. The bars are |∂h₈/∂hₜ| on a log scale: at the default w = 0.9 the states are 0.83, 0.42, 0.08, 0.85, 0.44, 0.09, −0.21, 0.77, and h₈ depends on h₁ with weight only 0.0335 (average factor 0.616 per step). At w = 0.5 it falls to 8.8e−4. At w = 2.5 the states saturate near 0.97 and the dependence collapses to 4.7e−9. Choose one → many and w = 1.5 to see bars above 1 (up to 2.15): the start of an exploding gradient.

## Input and output topologies

The same cell supports several sequence tasks. What changes is where inputs enter and where losses are attached. Karpathy's 2015 blog post *The unreasonable effectiveness of recurrent neural networks* popularised the diagram:

Where inputs enter and outputs leave the unrolled chain

| topology | inputs | outputs | typical task |
| --- | --- | --- | --- |
| one → many | one input (at $t=1$, or as $h_0$) | every step | image captioning, music or text generation from a prompt |
| many → one | every step | only $y_T$ | sentiment, spam, sequence classification |
| many → many, synced | every step | every step, aligned with $x_t$ | part-of-speech tagging, video frame labelling, language modelling |
| many → many, enc–dec | first phase | second phase, own length | translation, summarisation, speech recognition |

In generation (one → many and the decoder half of enc–dec) the input at step $t$ is usually the token produced at step $t-1$, so the model consumes its own output. During training one normally feeds the ground-truth previous token instead (see **Teacher forcing**). The unsynced case is the subject of **Encoder–decoder architecture** and **Sequence-to-sequence (Seq2Seq)**; when outputs may use future inputs too, see **Bidirectional RNN**.

## Backpropagation through time

With a loss at every output, $L=\sum_t L_t(y_t)$, the gradient for a shared weight sums its contribution at every step. The contribution of step $k$ to the loss at step $t$ travels back through every intermediate state:

$$
\frac{\partial L}{\partial \theta}=\sum_{t=1}^{T}\sum_{k=1}^{t}\frac{\partial L_t}{\partial h_t}\;\Bigg(\prod_{j=k+1}^{t}\frac{\partial h_j}{\partial h_{j-1}}\Bigg)\;\frac{\partial^{+} h_k}{\partial \theta}
$$

- $\partial h_j/\partial h_{j-1}$: the cell's local Jacobian at step $j$; for a vanilla RNN $\operatorname{diag}(1-h_j^2)\,W_{hh}$
- $\prod_{j=k+1}^{t}$: a product of $t-k$ Jacobians: the path from step $k$ to step $t$
- $\partial^{+}h_k/\partial\theta$: the "immediate" derivative of $h_k$ with $h_{k-1}$ held fixed

In practice autograd does this for you: run the forward loop, store every $h_t$, call `backward()`. The cost is one backward step per forward step, and memory grows with $T$ because every state must be kept. For long streams one cuts the graph every $k$ steps (see **Truncated backpropagation through time (TBPTT)**).

## Why long lags are hard

The product of Jacobians is the crux. If each factor has norm about $\gamma$, a dependence across lag $t-k$ scales like $\gamma^{\,t-k}$. With $\gamma<1$ it **vanishes** exponentially, so the loss at step 100 says nothing useful to the weights about the input at step 1. With $\gamma>1$ it **explodes**, and one update can throw the weights far away. A squashing nonlinearity such as tanh makes things worse in one direction: its slope $1-h^2$ goes to 0 when units saturate, so the figure's large-$w$ setting, which pushes $h$ toward $\pm1$, kills the gradient instead of exploding it. Bengio et al. (1994) proved the dilemma for a broad class of systems: to store information robustly against noise, the dynamics must be contracting, and contracting dynamics make gradients vanish. Details and remedies are in **Vanilla RNN**, **Vanishing gradients**, **Exploding gradients** and **Gradient clipping**.

> **Gates are the fix:** The **Long short-term memory (LSTM)** and **Gated recurrent unit (GRU)** replace the pure multiplication $h_t=\tanh(W h_{t-1}+\dots)$ with an *additive*, gated update such as $c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$. When the forget gate is near 1, the local Jacobian of the cell state is near the identity and gradients survive hundreds of steps. Almost every practical RNN since the late 1990s is a gated one.

## Worked example: four steps of a one-unit RNN

Take the figure's defaults $w=0.9$, $u=1.5$, $b=-0.3$, $h_0=0$, inputs $x=(1,0,0,1)$. Each step computes $a_t=w\,h_{t-1}+u\,x_t+b$, then $h_t=\tanh a_t$ and the local Jacobian $\partial h_t/\partial h_{t-1}=w(1-h_t^2)$:

Forward pass and local Jacobians (computed in Node from the figure's code)

| t | x_t | h_{t−1} | a_t | h_t | w(1 − h_t²) |
| --- | --- | --- | --- | --- | --- |
| 1 | 1 | 0 | 1.2000 | 0.8337 | 0.2745 |
| 2 | 0 | 0.8337 | 0.4503 | 0.4221 | 0.7396 |
| 3 | 0 | 0.4221 | 0.0799 | 0.0798 | 0.8943 |
| 4 | 1 | 0.0798 | 1.2718 | 0.8543 | 0.2432 |

**How much does h₄ depend on h₁?**

1. Chain the local Jacobians after step 1: $\partial h_4/\partial h_1=0.7396\times0.8943\times0.2432=0.1608$.
2. Check by finite differences: nudging $h_1$ by $\pm10^{-6}$ and rerunning steps 2–4 changes $h_4$ by $0.1608$ times the nudge.
3. The two steps that received an input of 1 (t = 1 and 4) have large $|a_t|$, so $h_t$ is closer to saturation and their factor is small (0.27, 0.24). Steps with no input stay near the linear region of tanh and pass about $w=0.9$.
4. Running all 8 steps of the figure gives $\partial h_8/\partial h_1=0.0335$: a lag of 7 steps already removes 97% of the signal.

```python
import torch, torch.nn as nn

N, T, I, H, C = 4, 12, 8, 32, 5           # batch, time, input, hidden, classes
x = torch.randn(N, T, I)

# 1) The loop, explicitly: one cell, same weights at every step
cell = nn.RNNCell(I, H, nonlinearity="tanh")
h = torch.zeros(N, H)
states = []
for t in range(T):
    h = cell(x[:, t], h)                  # h_t = tanh(W_ih x_t + b_ih + W_hh h_{t-1} + b_hh)
    states.append(h)
hs = torch.stack(states, dim=1)           # (N, T, H)

# 2) The same thing fused (cuDNN on GPU)
rnn = nn.RNN(I, H, batch_first=True)
out, h_n = rnn(x)                         # out: (N, T, H) all states; h_n: (1, N, H) last state

# many -> one: classify the sequence from the last state
head = nn.Linear(H, C)
logits_seq = head(h_n[-1])                # (N, C)

# many -> many (synced): one prediction per step
logits_tok = head(out)                    # (N, T, C)
y = torch.randint(0, C, (N, T))
loss = nn.functional.cross_entropy(logits_tok.reshape(-1, C), y.reshape(-1))
loss.backward()                           # backpropagation through time, done by autograd
print(sum(p.numel() for p in rnn.parameters()))   # H*(I+H+2) = 32*42 = 1344, independent of T
```

The recurrence by hand with nn.RNNCell, then the fused nn.RNN, in two topologies

> **Variable-length batches:** Sequences in a batch are padded to the same length. If you read the "last" state with `out[:, -1]` you get the state after the padding, not after the real last token. Either index each sequence's true last step, or pack the batch with `nn.utils.rnn.pack_padded_sequence`, in which case `h_n` is the state at each sequence's true end.

> **Where RNNs stand today:** Since 2017 Transformers (see **Self-Attention**) have replaced RNNs for most large-scale language tasks: attention connects any two positions in one step and trains in parallel across time. Recurrence has come back in a different form: **State space models (SSMs)** and **Mamba** are linear recurrences that train in parallel and still run with constant memory per token at inference. Convolutional alternatives are covered in **Temporal convolutional network (TCN)**; fixed random recurrences in **Echo state network**.

**Strengths**

- Handles sequences of any length with a fixed number of parameters
- Constant memory and $O(1)$ work per new step at inference: natural for streaming
- Weight sharing gives a built-in notion of position-independent patterns
- Simple, well-understood building block; gated variants (LSTM, GRU) remain strong on small and medium data

**Limitations**

- Sequential in time: step $t$ needs step $t-1$, so training does not parallelise over $T$
- Vanishing and exploding gradients make long-range dependencies hard to learn
- Everything about the past must squeeze through the fixed-size state $h_t$
- BPTT memory grows with $T$; long streams need truncation

**Question:** A vanilla RNN with input size 8 and hidden size 32 is applied to sequences of length 10 and then of length 1000. How does its parameter count change?

- A. It stays the same
- B. It grows 100 times
- C. It grows by the extra 990 hidden states
- D. It doubles because of BPTT

**Answer:** A. It stays the same — The same $W_{xh}$, $W_{hh}$ and biases are used at every step, so the count is $H(I+H+2)=1344$ for any $T$. What grows with $T$ is compute and the memory for the stored activations during training.

**Question:** Each local Jacobian of a one-unit RNN has magnitude about 0.6. Roughly how large is ∂h₂₁/∂h₁?

- A. About 0.6
- B. About 0.6 × 20 = 12
- C. About 0.6²⁰ ≈ 4e−5
- D. Exactly 1, because the weights are shared

**Answer:** C. About 0.6²⁰ ≈ 4e−5 — BPTT multiplies one factor per step, so a lag of 20 gives $0.6^{20}\approx3.7\times10^{-5}$. This exponential decay is the vanishing-gradient problem of Hochreiter (1991) and Bengio et al. (1994).

**Key points**

- An RNN updates a hidden state with a shared cell: $h_t=f_\theta(h_{t-1},x_t)$, optionally $y_t=g_\theta(h_t)$.
- Unrolled over $T$ steps it is a $T$-layer network with tied weights; parameters do not depend on $T$.
- Topologies: one → many, many → one, many → many synced, and encoder–decoder for unaligned lengths.
- Training is backpropagation through time; the gradient across a lag is a product of local Jacobians, so it vanishes or explodes geometrically.
- History: Jordan (1986) and Elman (1990) cells; Hochreiter (1991) and Bengio et al. (1994) diagnosed the long-lag problem, leading to LSTM and GRU.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
