Definition
Recurrent neural network (RNN) is a neural network with a loop. At every time step it combines the current input with its own previous hidden state to produce a new state , and optionally an output . The same parameters are used at every step, so one small cell can process a sequence of any length.
A feed-forward network maps one fixed-size input to one output. Text, speech, sensor streams and actions arrive as sequences whose length varies and whose meaning depends on order: "dog bites man" is not "man bites dog". You could concatenate a fixed window of inputs, but then the window caps how far back the model can look, and a pattern learned at position 3 has to be relearned at position 30. A recurrent network fixes both problems with one idea: keep a running summary of everything seen so far and update it with the same function at each step.
The idea grew out of the connectionist work of the 1980s. Hopfield (1982) used recurrent connections for associative memory (see Hopfield network), but those nets settle to fixed points rather than process sequences. Jordan (1986) fed the network's outputs back as extra "state" inputs to produce serial order. Elman (1990), Finding structure in time, fed back the hidden layer through "context units"; trained on next-word prediction, his network discovered word classes such as nouns and verbs from order alone. That hidden-to-hidden loop is the cell now called the simple or vanilla RNN. Rumelhart, Hinton and Williams (1986) had already shown how to train such nets by unfolding them in time, and Werbos (1990) gave the method its standard name, backpropagation through time (BPTT). The trouble came next. Hochreiter's 1991 diploma thesis and Bengio, Simard and Frasconi (1994), Learning long-term dependencies with gradient descent is difficult, showed that the gradient signal decays (or explodes) exponentially with the time lag, which motivated the LSTM (1997) and every gated design since.
The recurrence
- input at step (a word embedding, an audio frame, a sensor reading)
- hidden state: a fixed-size vector that summarises
- the cell. Vanilla RNN: ; LSTM and GRU add gates
- readout, e.g. logits followed by a softmax
- shared over time: the same weights at and
Unrolling the loop for steps turns the RNN into a -layer feed-forward network in which every layer has the same weights: . Two consequences follow. The number of parameters does not depend on , so the model generalises across positions and lengths. And training it means backpropagating through a network as deep as the sequence is long.
At a glance
- Input
- a sequence of vectors, any length
- State
- a fixed-size carried from step to step
- Parameters
- independent of (weight sharing over time)
- Compute
- cell evaluations, inherently sequential in
- Memory at inference
- : only the current state is kept, however long the stream
- Training
- backpropagation through time; memory for the stored states
- Main weakness
- gradients across long lags vanish or explode, so plain RNNs struggle beyond tens of steps
Input and output topologies
The same cell supports several sequence tasks. What changes is where inputs enter and where losses are attached. Karpathy's 2015 blog post The unreasonable effectiveness of recurrent neural networks popularised the diagram:
| topology | inputs | outputs | typical task |
|---|---|---|---|
| one → many | one input (at , or as ) | every step | image captioning, music or text generation from a prompt |
| many → one | every step | only | sentiment, spam, sequence classification |
| many → many, synced | every step | every step, aligned with | part-of-speech tagging, video frame labelling, language modelling |
| many → many, enc–dec | first phase | second phase, own length | translation, summarisation, speech recognition |
In generation (one → many and the decoder half of enc–dec) the input at step is usually the token produced at step , so the model consumes its own output. During training one normally feeds the ground-truth previous token instead (see Teacher forcing). The unsynced case is the subject of Encoder–decoder architecture and Sequence-to-sequence (Seq2Seq); when outputs may use future inputs too, see Bidirectional RNN.
Backpropagation through time
With a loss at every output, , the gradient for a shared weight sums its contribution at every step. The contribution of step to the loss at step travels back through every intermediate state:
- the cell's local Jacobian at step ; for a vanilla RNN
- a product of Jacobians: the path from step to step
- the "immediate" derivative of with held fixed
In practice autograd does this for you: run the forward loop, store every , call backward(). The cost is one backward step per forward step, and memory grows with because every state must be kept. For long streams one cuts the graph every steps (see Truncated backpropagation through time (TBPTT)).
Why long lags are hard
The product of Jacobians is the crux. If each factor has norm about , a dependence across lag scales like . With it vanishes exponentially, so the loss at step 100 says nothing useful to the weights about the input at step 1. With it explodes, and one update can throw the weights far away. A squashing nonlinearity such as tanh makes things worse in one direction: its slope goes to 0 when units saturate, so the figure's large- setting, which pushes toward , kills the gradient instead of exploding it. Bengio et al. (1994) proved the dilemma for a broad class of systems: to store information robustly against noise, the dynamics must be contracting, and contracting dynamics make gradients vanish. Details and remedies are in Vanilla RNN, Vanishing gradients, Exploding gradients and Gradient clipping.
Worked example: four steps of a one-unit RNN
Take the figure's defaults , , , , inputs . Each step computes , then and the local Jacobian :
| t | x_t | h_{t−1} | a_t | h_t | w(1 − h_t²) |
|---|---|---|---|---|---|
| 1 | 1 | 0 | 1.2000 | 0.8337 | 0.2745 |
| 2 | 0 | 0.8337 | 0.4503 | 0.4221 | 0.7396 |
| 3 | 0 | 0.4221 | 0.0799 | 0.0798 | 0.8943 |
| 4 | 1 | 0.0798 | 1.2718 | 0.8543 | 0.2432 |
How much does h₄ depend on h₁?
- 1Chain the local Jacobians after step 1: .
- 2Check by finite differences: nudging by and rerunning steps 2–4 changes by times the nudge.
- 3The two steps that received an input of 1 (t = 1 and 4) have large , so is closer to saturation and their factor is small (0.27, 0.24). Steps with no input stay near the linear region of tanh and pass about .
- 4Running all 8 steps of the figure gives : a lag of 7 steps already removes 97% of the signal.
1import torch, torch.nn as nn23N, T, I, H, C = 4, 12, 8, 32, 5 # batch, time, input, hidden, classes4x = torch.randn(N, T, I)56# 1) The loop, explicitly: one cell, same weights at every step7cell = nn.RNNCell(I, H, nonlinearity="tanh")8h = torch.zeros(N, H)9states = []10for t in range(T):11 h = cell(x[:, t], h) # h_t = tanh(W_ih x_t + b_ih + W_hh h_{t-1} + b_hh)12 states.append(h)13hs = torch.stack(states, dim=1) # (N, T, H)1415# 2) The same thing fused (cuDNN on GPU)16rnn = nn.RNN(I, H, batch_first=True)17out, h_n = rnn(x) # out: (N, T, H) all states; h_n: (1, N, H) last state1819# many -> one: classify the sequence from the last state20head = nn.Linear(H, C)21logits_seq = head(h_n[-1]) # (N, C)2223# many -> many (synced): one prediction per step24logits_tok = head(out) # (N, T, C)25y = torch.randint(0, C, (N, T))26loss = nn.functional.cross_entropy(logits_tok.reshape(-1, C), y.reshape(-1))27loss.backward() # backpropagation through time, done by autograd28print(sum(p.numel() for p in rnn.parameters())) # H*(I+H+2) = 32*42 = 1344, independent of TUse when
- +Handles sequences of any length with a fixed number of parameters
- +Constant memory and work per new step at inference: natural for streaming
- +Weight sharing gives a built-in notion of position-independent patterns
- +Simple, well-understood building block; gated variants (LSTM, GRU) remain strong on small and medium data
Avoid when
- −Sequential in time: step needs step , so training does not parallelise over
- −Vanishing and exploding gradients make long-range dependencies hard to learn
- −Everything about the past must squeeze through the fixed-size state
- −BPTT memory grows with ; long streams need truncation
Check yourself
A vanilla RNN with input size 8 and hidden size 32 is applied to sequences of length 10 and then of length 1000. How does its parameter count change?
Check yourself
Each local Jacobian of a one-unit RNN has magnitude about 0.6. Roughly how large is ∂h₂₁/∂h₁?
Remember
- ✓An RNN updates a hidden state with a shared cell: , optionally .
- ✓Unrolled over steps it is a -layer network with tied weights; parameters do not depend on .
- ✓Topologies: one → many, many → one, many → many synced, and encoder–decoder for unaligned lengths.
- ✓Training is backpropagation through time; the gradient across a lag is a product of local Jacobians, so it vanishes or explodes geometrically.
- ✓History: Jordan (1986) and Elman (1990) cells; Hochreiter (1991) and Bengio et al. (1994) diagnosed the long-lag problem, leading to LSTM and GRU.