Recurrent neural network (RNN)

A network that reads a sequence one step at a time and carries a hidden state, h_t = f(h_{t−1}, x_t), with the same weights at every step. Unrolled in time it is a deep net trained by backpropagation through time, whose gradients shrink or blow up with the time lag.

12 min read1115 wordscore0 viewsupdated 2026-10-12

Definition

Recurrent neural network (RNN) is a neural network with a loop. At every time step tt it combines the current input xtx_t with its own previous hidden state ht−1h_{t-1} to produce a new state ht=fθ(ht−1,xt)h_t=f_\theta(h_{t-1},x_t), and optionally an output yt=gθ(ht)y_t=g_\theta(h_t). The same parameters θ\theta are used at every step, so one small cell can process a sequence of any length.

A feed-forward network maps one fixed-size input to one output. Text, speech, sensor streams and actions arrive as sequences whose length varies and whose meaning depends on order: "dog bites man" is not "man bites dog". You could concatenate a fixed window of inputs, but then the window caps how far back the model can look, and a pattern learned at position 3 has to be relearned at position 30. A recurrent network fixes both problems with one idea: keep a running summary hth_t of everything seen so far and update it with the same function at each step.

The idea grew out of the connectionist work of the 1980s. Hopfield (1982) used recurrent connections for associative memory (see Hopfield network), but those nets settle to fixed points rather than process sequences. Jordan (1986) fed the network's outputs back as extra "state" inputs to produce serial order. Elman (1990), Finding structure in time, fed back the hidden layer through "context units"; trained on next-word prediction, his network discovered word classes such as nouns and verbs from order alone. That hidden-to-hidden loop is the cell now called the simple or vanilla RNN. Rumelhart, Hinton and Williams (1986) had already shown how to train such nets by unfolding them in time, and Werbos (1990) gave the method its standard name, backpropagation through time (BPTT). The trouble came next. Hochreiter's 1991 diploma thesis and Bengio, Simard and Frasconi (1994), Learning long-term dependencies with gradient descent is difficult, showed that the gradient signal decays (or explodes) exponentially with the time lag, which motivated the LSTM (1997) and every gated design since.

The recurrence

ht=fθ(ht−1, xt),yt=gθ(ht),h0=given (usually 0)h_t=f_\theta\big(h_{t-1},\,x_t\big),\qquad y_t=g_\theta(h_t),\qquad h_0=\text{given (usually }0)
xtx_t
input at step tt (a word embedding, an audio frame, a sensor reading)
hth_t
hidden state: a fixed-size vector that summarises x1,…,xtx_1,\dots,x_t
fθf_\theta
the cell. Vanilla RNN: tanh⁡(Wxhxt+Whhht−1+b)\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b); LSTM and GRU add gates
gθg_\theta
readout, e.g. logits Whyht+byW_{hy}h_t+b_y followed by a softmax
θ\theta
shared over time: the same weights at t=1t=1 and t=1000t=1000

Unrolling the loop for TT steps turns the RNN into a TT-layer feed-forward network in which every layer has the same weights: hT=f(f(⋯f(h0,x1)⋯ ,xT−1),xT)h_T=f(f(\cdots f(h_0,x_1)\cdots,x_{T-1}),x_T). Two consequences follow. The number of parameters does not depend on TT, so the model generalises across positions and lengths. And training it means backpropagating through a network as deep as the sequence is long.

At a glance

Input
a sequence x1,…,xTx_1,\dots,x_T of vectors, any length TT
State
a fixed-size ht∈RHh_t\in\mathbb R^H carried from step to step
Parameters
independent of TT (weight sharing over time)
Compute
O(T)O(T) cell evaluations, inherently sequential in tt
Memory at inference
O(H)O(H): only the current state is kept, however long the stream
Training
backpropagation through time; memory O(T⋅H)O(T\cdot H) for the stored states
Main weakness
gradients across long lags vanish or explode, so plain RNNs struggle beyond tens of steps
hₜ = tanh(w·hₜ₋₁ + u·xₜ + b), yₜ = v·hₜ · same w, u, b, v at every stepfoldedhwxy=h10.831t=1wh20.420t=2wh30.080t=3wh40.851t=4wh50.440t=5wh60.090t=6wh7−0.210t=7wh80.771t=80.77h2 = tanh(0.90·0.834 + 1.50·0 − 0.3) = tanh(0.450) = 0.422local Jacobian ∂h2/∂h1 = w·(1 − h2²) = 0.74011e−21e−41e−6|∂h8/∂hₜ|(log scale)0.0330.0450.0510.2080.2860.3210.3731.000many → one: sentiment, sequence classificationBPTT: ∂h8/∂h₁ = 0.033 · mean |local Jacobian| per step = 0.616 → vanishing
fig.A one-unit RNN, hₜ = tanh(w·hₜ₋₁ + 1.5·xₜ − 0.3), folded (left) and unrolled over 8 steps on the input bits 1 0 0 1 0 0 0 1. Each box shows hₜ, and the same w sits on every arrow. Click a step to see its update written out. The topology buttons change where inputs are fed and outputs are read. The bars are |∂h₈/∂hₜ| on a log scale: at the default w = 0.9 the states are 0.83, 0.42, 0.08, 0.85, 0.44, 0.09, −0.21, 0.77, and h₈ depends on h₁ with weight only 0.0335 (average factor 0.616 per step). At w = 0.5 it falls to 8.8e−4. At w = 2.5 the states saturate near 0.97 and the dependence collapses to 4.7e−9. Choose one → many and w = 1.5 to see bars above 1 (up to 2.15): the start of an exploding gradient.

Input and output topologies

The same cell supports several sequence tasks. What changes is where inputs enter and where losses are attached. Karpathy's 2015 blog post The unreasonable effectiveness of recurrent neural networks popularised the diagram:

topologyinputsoutputstypical task
one → manyone input (at t=1t=1, or as h0h_0)every stepimage captioning, music or text generation from a prompt
many → oneevery steponly yTy_Tsentiment, spam, sequence classification
many → many, syncedevery stepevery step, aligned with xtx_tpart-of-speech tagging, video frame labelling, language modelling
many → many, enc–decfirst phasesecond phase, own lengthtranslation, summarisation, speech recognition
Where inputs enter and outputs leave the unrolled chain

In generation (one → many and the decoder half of enc–dec) the input at step tt is usually the token produced at step t−1t-1, so the model consumes its own output. During training one normally feeds the ground-truth previous token instead (see Teacher forcing). The unsynced case is the subject of Encoder–decoder architecture and Sequence-to-sequence (Seq2Seq); when outputs may use future inputs too, see Bidirectional RNN.

Backpropagation through time

With a loss at every output, L=∑tLt(yt)L=\sum_t L_t(y_t), the gradient for a shared weight sums its contribution at every step. The contribution of step kk to the loss at step tt travels back through every intermediate state:

∂L∂θ=∑t=1T∑k=1t∂Lt∂ht  (∏j=k+1t∂hj∂hj−1)  ∂+hk∂θ\frac{\partial L}{\partial \theta}=\sum_{t=1}^{T}\sum_{k=1}^{t}\frac{\partial L_t}{\partial h_t}\;\Bigg(\prod_{j=k+1}^{t}\frac{\partial h_j}{\partial h_{j-1}}\Bigg)\;\frac{\partial^{+} h_k}{\partial \theta}
∂hj/∂hj−1\partial h_j/\partial h_{j-1}
the cell's local Jacobian at step jj; for a vanilla RNN diag⁡(1−hj2) Whh\operatorname{diag}(1-h_j^2)\,W_{hh}
∏j=k+1t\prod_{j=k+1}^{t}
a product of t−kt-k Jacobians: the path from step kk to step tt
∂+hk/∂θ\partial^{+}h_k/\partial\theta
the "immediate" derivative of hkh_k with hk−1h_{k-1} held fixed

In practice autograd does this for you: run the forward loop, store every hth_t, call backward(). The cost is one backward step per forward step, and memory grows with TT because every state must be kept. For long streams one cuts the graph every kk steps (see Truncated backpropagation through time (TBPTT)).

Why long lags are hard

The product of Jacobians is the crux. If each factor has norm about γ\gamma, a dependence across lag t−kt-k scales like γ t−k\gamma^{\,t-k}. With γ<1\gamma<1 it vanishes exponentially, so the loss at step 100 says nothing useful to the weights about the input at step 1. With γ>1\gamma>1 it explodes, and one update can throw the weights far away. A squashing nonlinearity such as tanh makes things worse in one direction: its slope 1−h21-h^2 goes to 0 when units saturate, so the figure's large-ww setting, which pushes hh toward ±1\pm1, kills the gradient instead of exploding it. Bengio et al. (1994) proved the dilemma for a broad class of systems: to store information robustly against noise, the dynamics must be contracting, and contracting dynamics make gradients vanish. Details and remedies are in Vanilla RNN, Vanishing gradients, Exploding gradients and Gradient clipping.

Worked example: four steps of a one-unit RNN

Take the figure's defaults w=0.9w=0.9, u=1.5u=1.5, b=−0.3b=-0.3, h0=0h_0=0, inputs x=(1,0,0,1)x=(1,0,0,1). Each step computes at=w ht−1+u xt+ba_t=w\,h_{t-1}+u\,x_t+b, then ht=tanh⁡ath_t=\tanh a_t and the local Jacobian ∂ht/∂ht−1=w(1−ht2)\partial h_t/\partial h_{t-1}=w(1-h_t^2):

tx_th_{t−1}a_th_tw(1 − h_t²)
1101.20000.83370.2745
200.83370.45030.42210.7396
300.42210.07990.07980.8943
410.07981.27180.85430.2432
Forward pass and local Jacobians (computed in Node from the figure's code)

How much does h₄ depend on h₁?

  1. 1Chain the local Jacobians after step 1: ∂h4/∂h1=0.7396×0.8943×0.2432=0.1608\partial h_4/\partial h_1=0.7396\times0.8943\times0.2432=0.1608.
  2. 2Check by finite differences: nudging h1h_1 by ±10−6\pm10^{-6} and rerunning steps 2–4 changes h4h_4 by 0.16080.1608 times the nudge.
  3. 3The two steps that received an input of 1 (t = 1 and 4) have large ∣at∣|a_t|, so hth_t is closer to saturation and their factor is small (0.27, 0.24). Steps with no input stay near the linear region of tanh and pass about w=0.9w=0.9.
  4. 4Running all 8 steps of the figure gives ∂h8/∂h1=0.0335\partial h_8/\partial h_1=0.0335: a lag of 7 steps already removes 97% of the signal.
python
1import torch, torch.nn as nn
2
3N, T, I, H, C = 4, 12, 8, 32, 5 # batch, time, input, hidden, classes
4x = torch.randn(N, T, I)
5
6# 1) The loop, explicitly: one cell, same weights at every step
7cell = nn.RNNCell(I, H, nonlinearity="tanh")
8h = torch.zeros(N, H)
9states = []
10for t in range(T):
11 h = cell(x[:, t], h) # h_t = tanh(W_ih x_t + b_ih + W_hh h_{t-1} + b_hh)
12 states.append(h)
13hs = torch.stack(states, dim=1) # (N, T, H)
14
15# 2) The same thing fused (cuDNN on GPU)
16rnn = nn.RNN(I, H, batch_first=True)
17out, h_n = rnn(x) # out: (N, T, H) all states; h_n: (1, N, H) last state
18
19# many -> one: classify the sequence from the last state
20head = nn.Linear(H, C)
21logits_seq = head(h_n[-1]) # (N, C)
22
23# many -> many (synced): one prediction per step
24logits_tok = head(out) # (N, T, C)
25y = torch.randint(0, C, (N, T))
26loss = nn.functional.cross_entropy(logits_tok.reshape(-1, C), y.reshape(-1))
27loss.backward() # backpropagation through time, done by autograd
28print(sum(p.numel() for p in rnn.parameters())) # H*(I+H+2) = 32*42 = 1344, independent of T
The recurrence by hand with nn.RNNCell, then the fused nn.RNN, in two topologies

Use when

  • +Handles sequences of any length with a fixed number of parameters
  • +Constant memory and O(1)O(1) work per new step at inference: natural for streaming
  • +Weight sharing gives a built-in notion of position-independent patterns
  • +Simple, well-understood building block; gated variants (LSTM, GRU) remain strong on small and medium data

Avoid when

  • −Sequential in time: step tt needs step t−1t-1, so training does not parallelise over TT
  • −Vanishing and exploding gradients make long-range dependencies hard to learn
  • −Everything about the past must squeeze through the fixed-size state hth_t
  • −BPTT memory grows with TT; long streams need truncation

Check yourself

A vanilla RNN with input size 8 and hidden size 32 is applied to sequences of length 10 and then of length 1000. How does its parameter count change?

Check yourself

Each local Jacobian of a one-unit RNN has magnitude about 0.6. Roughly how large is ∂h₂₁/∂h₁?

Remember

  • ✓An RNN updates a hidden state with a shared cell: ht=fθ(ht−1,xt)h_t=f_\theta(h_{t-1},x_t), optionally yt=gθ(ht)y_t=g_\theta(h_t).
  • ✓Unrolled over TT steps it is a TT-layer network with tied weights; parameters do not depend on TT.
  • ✓Topologies: one → many, many → one, many → many synced, and encoder–decoder for unaligned lengths.
  • ✓Training is backpropagation through time; the gradient across a lag is a product of local Jacobians, so it vanishes or explodes geometrically.
  • ✓History: Jordan (1986) and Elman (1990) cells; Hochreiter (1991) and Bengio et al. (1994) diagnosed the long-lag problem, leading to LSTM and GRU.
← back to Deep Learning roadmap