Definition
Long short-term memory (LSTM) is a recurrent cell that keeps two states: a cell state , the memory, and a hidden state , the output. Three sigmoid gates decide how the memory changes at each step: the input gate (how much new content to write), the forget gate (how much old memory to keep) and the output gate (how much of the memory to expose). The memory is updated by addition, , not by squashing it through a matrix and a nonlinearity at every step.
A vanilla RNN rewrites its whole state every step, . Backpropagating through steps multiplies Jacobians , and that product shrinks or blows up exponentially in (see Recurrent neural network (RNN) and Vanilla RNN). In practice a tanh RNN trained by BPTT struggles to use information more than about 10 to 20 steps old. The LSTM was designed to remove that limit: give the network a path through time on which the error signal is multiplied by (close to) 1 at every step, and let learned gates decide when to use it.
Sepp Hochreiter analysed the vanishing gradient in his 1991 diploma thesis (supervised by Jürgen Schmidhuber), and Bengio, Simard and Frasconi (1994) showed the same difficulty independently. Hochreiter and Schmidhuber's "Long Short-Term Memory" (Neural Computation 9(8), 1997) introduced the memory cell with a self-connection of fixed weight 1.0, the constant error carousel (CEC), protected by multiplicative input and output gates. They reported learning tasks with minimal time lags of more than 1000 steps, far beyond the RNNs of the time. The 1997 cell had no forget gate. Gers, Schmidhuber and Cummins ("Learning to Forget", Neural Computation 2000) added it, because on long continual input streams without resets the cell state could grow without bound. Gers and Schmidhuber (2000) also added peephole connections from the cell to the gates, for precise timing. The form used today, with forget gate and trained by full backpropagation through time, is the one of Graves and Schmidhuber (2005).
The gate equations
- input, hidden state and cell state; unless given
- logistic sigmoid, so every gate value is a per-unit fraction in
- the candidate content, in ; PyTorch's documentation calls it the "cell gate"
- element-wise product: each of the H memory units has its own gates
- PyTorch keeps two bias vectors (for cuDNN compatibility); only their sum matters
All four pre-activations are affine in the same pair , so implementations compute them with one matrix product of width and split the result. In PyTorch the rows of weight_ih_l0 (shape ) and weight_hh_l0 () are packed in the order : input, forget, cell (candidate), output. Keras uses the same order.
At a glance
- States
- cell (memory, unbounded in principle) and hidden
- Gates
- input , forget , output (sigmoid), plus the tanh candidate : four blocks of units
- Parameters (PyTorch)
- per layer;
nn.LSTM(10, 20)has - vs other cells
- a vanilla RNN of the same ( for ), a GRU ()
- Gradient along the cell
- plus indirect terms through
- Common init
- forget-gate bias (Jozefowicz et al., 2015; Keras
unit_forget_bias=True) - Not in nn.LSTM
- peepholes, layer norm inside the cell;
proj_sizeadds an LSTMP projection (Sak et al., 2014)
The cell state is a conveyor belt
Read the update line by line. The forget gate scales the old memory, the input gate scales a new candidate, and the two are added. If and the memory passes through unchanged, however many steps that lasts. If and it is overwritten. Anything in between blends the two, separately for each of the units. The output gate does not touch the memory at all. It only decides what the rest of the network sees through , so a cell can hold a value for a long time without letting it disturb the computation until it is needed.
Why the gradient survives: the constant error carousel
Differentiate the cell update with respect to the previous cell state. One term comes straight from the product . The others go through , which feeds every gate at step :
- the direct term: no weight matrix and no , just the forget gate
- ; the bracket also contains matrices, so these terms behave like a vanilla RNN's Jacobian
- how much of the error at step reaches step along the memory; close to 1 when the forget gates stay open
In the 1997 cell the forget gate did not exist, so the cell path had factor exactly 1 at every step: error entering the cell "rides the carousel" back through time without decay. Hochreiter and Schmidhuber also truncated the gradient wherever it left the cell, so only the CEC carried error across time. With a forget gate the factor is , which the network learns. Keeping preserves both the memory and its gradient. Dropping toward 0 forgets on purpose. The vanishing gradient is not abolished (the gate path still vanishes, and decays if the gates are not near 1), but the network can now choose to keep a long path open.
Forget-gate bias: start by remembering
At initialisation the weights are small, so each gate sits near . With a zero bias , and the cell path halves the gradient every step: after 100 steps. The network then has to learn to remember before it can learn anything long-range. Gers et al. (2000) already started their forget gates open, and Jozefowicz, Zaremba and Sutskever (ICML 2015), after evaluating more than ten thousand recurrent architectures, recommended a forget bias of 1: in their experiments it closed the gap between the LSTM and the GRU. A bias of 1 gives , which still decays fast over long lags (). It only makes remembering easier to learn. Tasks with very long dependencies often use larger values, and the figure shows why: , .
Worked example: one step of a one-unit LSTM
Take , input , previous states , , and weights per gate: , , , . The forget bias of 1.0 follows the usual initialisation.
| block | pre-activation | activation | value |
|---|---|---|---|
| input | |||
| forget | |||
| candidate | |||
| output |
Forward, then the gradient back to the previous cell state
- 1Memory update: . The cell state is not bounded by 1: it is a sum, not a tanh.
- 2Output: .
- 3Suppose . Then .
- 4Gate gradients: , , , and . Each is then multiplied by its activation's derivative to reach the weights.
- 5Along the cell path, : the error crosses the step multiplied by and nothing else.
1import torch, torch.nn as nn23torch.manual_seed(0)4N, L, I, H = 4, 7, 10, 205lstm = nn.LSTM(input_size=I, hidden_size=H, num_layers=1, batch_first=True)6x = torch.randn(N, L, I) # (batch, time, features) because batch_first=True78out, (h_n, c_n) = lstm(x) # h_0 and c_0 default to zeros9print(out.shape, h_n.shape, c_n.shape) # (4, 7, 20) (1, 4, 20) (1, 4, 20)10print(sum(p.numel() for p in lstm.parameters())) # 2560 = 4 * (H*I + H*H + 2*H)11print(lstm.weight_ih_l0.shape) # (80, 10): rows [i | f | g | o], H each1213# Forget-gate bias of 1 (the second block of four; the two bias vectors are summed)14with torch.no_grad():15 lstm.bias_ih_l0[H:2 * H].fill_(1.0)16 lstm.bias_hh_l0[H:2 * H].zero_()1718def step(x_t, h, c):19 z = x_t @ lstm.weight_ih_l0.T + lstm.bias_ih_l0 + h @ lstm.weight_hh_l0.T + lstm.bias_hh_l020 i, f, g, o = z.chunk(4, dim=-1)21 c = f.sigmoid() * c + i.sigmoid() * g.tanh()22 return o.sigmoid() * c.tanh(), c2324h = c = torch.zeros(N, H)25for t in range(L):26 h, c = step(x[:, t], h, c)27out, (h_n, c_n) = lstm(x)28print(torch.allclose(h, h_n[0], atol=1e-5), torch.allclose(c, c_n[0], atol=1e-5)) # True True29print(torch.allclose(out[:, -1], h_n[0])) # the last output is the final hidden state3031# The same step with the cell module, e.g. for custom loops or autoregressive decoding32cell = nn.LSTMCell(I, H)33h, c = cell(x[:, 0], (torch.zeros(N, H), torch.zeros(N, H)))Variants
| variant | change | where |
|---|---|---|
| Original (1997) | no forget gate: ; truncated gradient | Hochreiter & Schmidhuber |
| Forget gate | Gers, Schmidhuber & Cummins 2000; the standard today | |
| Peephole | gates also read the cell: , , | Gers & Schmidhuber 2000; not in nn.LSTM |
| Coupled gates (CIFG) | : write exactly as much as is forgotten | Greff et al. 2017; the idea behind the GRU's update gate |
| Projection (LSTMP) | with fewer rows than | Sak et al. 2014; proj_size in nn.LSTM |
Greff, Srivastava, Koutník, Steunebrink and Schmidhuber ("LSTM: A Search Space Odyssey", IEEE TNNLS 2017) compared eight variants on speech (TIMIT), handwriting (IAM Online) and polyphonic music (JSB Chorales) in about 5400 runs. None improved significantly on the standard cell with forget gate and peepholes, and the forget gate and the output activation (the tanh on ) were the most critical parts. Coupling the input and forget gates or removing peepholes did not hurt significantly, which is part of why the simpler GRU works as well as it does.
Use when
- +Learns dependencies over hundreds of steps: the cell path scales the gradient only by
- +Gates make memory content-dependent: write, keep and read are decided per unit and per step
- +Constant memory and compute per step at inference: well suited to streaming
- +Mature, robust and fused in cuDNN; works well with modest tuning
Avoid when
- −Sequential over time: step needs , so training does not parallelise across the sequence like attention
- − the parameters and FLOPs of a vanilla RNN, a GRU
- −The state is a fixed-size vector: very long contexts must be compressed into numbers
- −Still needs gradient clipping and careful dropout; the gate path still vanishes
Check yourself
A forget gate stays at f = 0.5 for 100 steps (zero bias, small weights). By what factor is the gradient along the cell path scaled between step 0 and step 100?
Check yourself
How many parameters does nn.LSTM(input_size=10, hidden_size=20) have?
Remember
- ✓Two states: memory and output ; input, forget and output gates are sigmoids of .
- ✓Along the cell path : no weight matrix, no . With memory and gradient survive (the constant error carousel).
- ✓Hochreiter & Schmidhuber (1997) had no forget gate; Gers et al. (2000) added it so cells can reset; peepholes are optional, and Greff et al. (2017) found no variant clearly better than the standard cell.
- ✓Initialise the forget bias positive (1 is the usual choice); in PyTorch it is the slice [H:2H] of the packed i, f, g, o rows, and the two bias vectors add.
- ✓Parameters per layer: four times a vanilla RNN, four thirds of a GRU.