Long short-term memory (LSTM)

A recurrent cell with a separate memory c_t, updated additively as c_t = f_t⊙c_{t−1} + i_t⊙g_t, and sigmoid gates that decide what to write, keep and read. Along the cell path the gradient is scaled only by the forget gate, so information and error survive hundreds of steps.

14 min read1378 wordscore0 viewsupdated 2026-10-12

Definition

Long short-term memory (LSTM) is a recurrent cell that keeps two states: a cell state ctc_t, the memory, and a hidden state hth_t, the output. Three sigmoid gates decide how the memory changes at each step: the input gate iti_t (how much new content to write), the forget gate ftf_t (how much old memory to keep) and the output gate oto_t (how much of the memory to expose). The memory is updated by addition, ct=ft⊙ct−1+it⊙gtc_t=f_t\odot c_{t-1}+i_t\odot g_t, not by squashing it through a matrix and a nonlinearity at every step.

A vanilla RNN rewrites its whole state every step, ht=tanh⁡(Wxhxt+Whhht−1+b)h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b). Backpropagating through kk steps multiplies kk Jacobians diag⁡(tanh⁡′) Whh\operatorname{diag}(\tanh')\,W_{hh}, and that product shrinks or blows up exponentially in kk (see Recurrent neural network (RNN) and Vanilla RNN). In practice a tanh RNN trained by BPTT struggles to use information more than about 10 to 20 steps old. The LSTM was designed to remove that limit: give the network a path through time on which the error signal is multiplied by (close to) 1 at every step, and let learned gates decide when to use it.

Sepp Hochreiter analysed the vanishing gradient in his 1991 diploma thesis (supervised by Jürgen Schmidhuber), and Bengio, Simard and Frasconi (1994) showed the same difficulty independently. Hochreiter and Schmidhuber's "Long Short-Term Memory" (Neural Computation 9(8), 1997) introduced the memory cell with a self-connection of fixed weight 1.0, the constant error carousel (CEC), protected by multiplicative input and output gates. They reported learning tasks with minimal time lags of more than 1000 steps, far beyond the RNNs of the time. The 1997 cell had no forget gate. Gers, Schmidhuber and Cummins ("Learning to Forget", Neural Computation 2000) added it, because on long continual input streams without resets the cell state could grow without bound. Gers and Schmidhuber (2000) also added peephole connections from the cell to the gates, for precise timing. The form used today, with forget gate and trained by full backpropagation through time, is the one of Graves and Schmidhuber (2005).

The gate equations

it=σ(Wiixt+bii+Whiht−1+bhi)input gateft=σ(Wifxt+bif+Whfht−1+bhf)forget gategt=tanh⁡(Wigxt+big+Whght−1+bhg)candidateot=σ(Wioxt+bio+Whoht−1+bho)output gatect=ft⊙ct−1+it⊙gtht=ot⊙tanh⁡(ct)\begin{aligned} i_t&=\sigma(W_{ii}x_t+b_{ii}+W_{hi}h_{t-1}+b_{hi}) && \text{input gate}\\ f_t&=\sigma(W_{if}x_t+b_{if}+W_{hf}h_{t-1}+b_{hf}) && \text{forget gate}\\ g_t&=\tanh(W_{ig}x_t+b_{ig}+W_{hg}h_{t-1}+b_{hg}) && \text{candidate}\\ o_t&=\sigma(W_{io}x_t+b_{io}+W_{ho}h_{t-1}+b_{ho}) && \text{output gate}\\ c_t&=f_t\odot c_{t-1}+i_t\odot g_t\\ h_t&=o_t\odot\tanh(c_t) \end{aligned}
xt∈RI, ht,ct∈RHx_t\in\mathbb R^{I},\ h_t,c_t\in\mathbb R^{H}
input, hidden state and cell state; h0=c0=0h_0=c_0=0 unless given
σ\sigma
logistic sigmoid, so every gate value is a per-unit fraction in (0,1)(0,1)
gtg_t
the candidate content, in (−1,1)(-1,1); PyTorch's documentation calls it the "cell gate"
⊙\odot
element-wise product: each of the H memory units has its own gates
bi⋅, bh⋅b_{i\cdot},\ b_{h\cdot}
PyTorch keeps two bias vectors (for cuDNN compatibility); only their sum matters

All four pre-activations are affine in the same pair (xt,ht−1)(x_t,h_{t-1}), so implementations compute them with one matrix product of width 4H4H and split the result. In PyTorch the rows of weight_ih_l0 (shape 4H×I4H\times I) and weight_hh_l0 (4H×H4H\times H) are packed in the order (W⋅i ∣ W⋅f ∣ W⋅g ∣ W⋅o)(W_{\cdot i}\,|\,W_{\cdot f}\,|\,W_{\cdot g}\,|\,W_{\cdot o}): input, forget, cell (candidate), output. Keras uses the same order.

At a glance

States
cell ctc_t (memory, unbounded in principle) and hidden ht=ot⊙tanh⁡(ct)∈(−1,1)h_t=o_t\odot\tanh(c_t)\in(-1,1)
Gates
input ii, forget ff, output oo (sigmoid), plus the tanh candidate gg: four blocks of HH units
Parameters (PyTorch)
4(HI+H2+2H)4\big(HI+H^2+2H\big) per layer; nn.LSTM(10, 20) has 4(200+400+40)=25604(200+400+40)=2560
vs other cells
4×4\times a vanilla RNN of the same HH (640640 for I=10,H=20I=10,H=20), 43×\tfrac43\times a GRU (19201920)
Gradient along the cell
∂ct/∂ct−1=diag⁡(ft)\partial c_t/\partial c_{t-1}=\operatorname{diag}(f_t) plus indirect terms through ht−1h_{t-1}
Common init
forget-gate bias +1+1 (Jozefowicz et al., 2015; Keras unit_forget_bias=True)
Not in nn.LSTM
peepholes, layer norm inside the cell; proj_size adds an LSTMP projection (Sak et al., 2014)

The cell state is a conveyor belt

Read the update line by line. The forget gate scales the old memory, the input gate scales a new candidate, and the two are added. If ft=1f_t=1 and it=0i_t=0 the memory passes through unchanged, however many steps that lasts. If ft=0f_t=0 and it=1i_t=1 it is overwritten. Anything in between blends the two, separately for each of the HH units. The output gate does not touch the memory at all. It only decides what the rest of the network sees through hth_t, so a cell can hold a value for a long time without letting it disturb the computation until it is needed.

cₜ = fₜ·cₜ₋₁ + iₜ·gₜ, hₜ = oₜ·tanh(cₜ) · write 0.8 at t=2, read at t=39xₜWRiₜfₜoₜ−101∂cT−1/∂cₛ along the cell path = Π fₖ (log scale)11e-21e-41e-61e-8t=0t=19t=39━ cell cₜ ┄ hidden hₜ ■ write ■ read ■ erase · gate strips: darker = closer to 1
t=39 x=−0.29i=0.018 f=0.731g=−0.523 o=0.982c=−0.012 h=−0.012stored c2=0.883 → read h39=−0.012∂c39/∂c2 = 0.73137 = 9.25e-6
σ = 0.731
fig.One LSTM unit with hand-set weights: the input gate opens only on the write cue at t = 2 (value 0.8, so c₂ ≈ 0.88), the output gate only on the read cue at the last step (t = 39), and in between come 36 random distractors. The forget gate is σ(b_f) at every step, so along the cell path ∂c₃₉/∂c₂ = σ(b_f)³⁷. At the default b_f = 1, σ = 0.731 and the product is 9.25e−6: the memory has decayed to distractor noise by the time it is read (h₃₉ = −0.012). Raise b_f to 5: σ = 0.993, the product is 0.780 and the read gives h₃₉ = 0.563. Switch the forget gate off (the 1997 cell, f ≡ 1): the product is exactly 1 and h₃₉ = 0.671, but the erase cue can no longer clear the memory. With the forget gate on, b_f = 5 and the erase cue, one step with f = 9.1e−4 wipes it (c₃₉ = 0.002).

Differentiate the cell update with respect to the previous cell state. One term comes straight from the product ft⊙ct−1f_t\odot c_{t-1}. The others go through ht−1=ot−1⊙tanh⁡(ct−1)h_{t-1}=o_{t-1}\odot\tanh(c_{t-1}), which feeds every gate at step tt:

∂ct∂ct−1=diag⁡(ft)⏟cell path+[diag⁡(ct−1)∂ft∂ht−1+diag⁡(gt)∂it∂ht−1+diag⁡(it)∂gt∂ht−1]∂ht−1∂ct−1⏟through the gates,∂cT∂cs∣cell path=∏k=s+1Tdiag⁡(fk)\frac{\partial c_t}{\partial c_{t-1}}=\underbrace{\operatorname{diag}(f_t)}_{\text{cell path}}+\underbrace{\Big[\operatorname{diag}(c_{t-1})\tfrac{\partial f_t}{\partial h_{t-1}}+\operatorname{diag}(g_t)\tfrac{\partial i_t}{\partial h_{t-1}}+\operatorname{diag}(i_t)\tfrac{\partial g_t}{\partial h_{t-1}}\Big]\tfrac{\partial h_{t-1}}{\partial c_{t-1}}}_{\text{through the gates}},\qquad \frac{\partial c_T}{\partial c_s}\Big|_{\text{cell path}}=\prod_{k=s+1}^{T}\operatorname{diag}(f_k)
diag⁡(ft)\operatorname{diag}(f_t)
the direct term: no weight matrix and no tanh⁡′\tanh', just the forget gate
∂ht−1∂ct−1\tfrac{\partial h_{t-1}}{\partial c_{t-1}}
diag⁡(ot−1⊙(1−tanh⁡2ct−1))\operatorname{diag}\big(o_{t-1}\odot(1-\tanh^2 c_{t-1})\big); the bracket also contains Wh⋅W_{h\cdot} matrices, so these terms behave like a vanilla RNN's Jacobian
∏kfk\prod_k f_k
how much of the error at step TT reaches step ss along the memory; close to 1 when the forget gates stay open

In the 1997 cell the forget gate did not exist, so the cell path had factor exactly 1 at every step: error entering the cell "rides the carousel" back through time without decay. Hochreiter and Schmidhuber also truncated the gradient wherever it left the cell, so only the CEC carried error across time. With a forget gate the factor is fkf_k, which the network learns. Keeping f≈1f\approx1 preserves both the memory and its gradient. Dropping ff toward 0 forgets on purpose. The vanishing gradient is not abolished (the gate path still vanishes, and ∏fk\prod f_k decays if the gates are not near 1), but the network can now choose to keep a long path open.

Forget-gate bias: start by remembering

At initialisation the weights are small, so each gate sits near σ(b)\sigma(b). With a zero bias f≈0.5f\approx0.5, and the cell path halves the gradient every step: 0.5100≈7.9×10−310.5^{100}\approx7.9\times10^{-31} after 100 steps. The network then has to learn to remember before it can learn anything long-range. Gers et al. (2000) already started their forget gates open, and Jozefowicz, Zaremba and Sutskever (ICML 2015), after evaluating more than ten thousand recurrent architectures, recommended a forget bias of 1: in their experiments it closed the gap between the LSTM and the GRU. A bias of 1 gives σ(1)=0.731\sigma(1)=0.731, which still decays fast over long lags (0.731100≈2.5×10−140.731^{100}\approx2.5\times10^{-14}). It only makes remembering easier to learn. Tasks with very long dependencies often use larger values, and the figure shows why: σ(3)100=0.0078\sigma(3)^{100}=0.0078, σ(5)100=0.51\sigma(5)^{100}=0.51.

Worked example: one step of a one-unit LSTM

Take I=H=1I=H=1, input xt=1x_t=1, previous states ht−1=0.5h_{t-1}=0.5, ct−1=0.8c_{t-1}=0.8, and weights (wx, wh, b)(w_x,\,w_h,\,b) per gate: i:(0.5,−0.3,0.1)i:(0.5,-0.3,0.1), f:(0.4,0.6,1.0)f:(0.4,0.6,1.0), g:(1.0,−0.5,0)g:(1.0,-0.5,0), o:(−0.2,0.8,0.2)o:(-0.2,0.8,0.2). The forget bias of 1.0 follows the usual initialisation.

blockpre-activationactivationvalue
input ii0.5−0.15+0.1=0.450.5-0.15+0.1=0.45σ\sigma0.61060.6106
forget ff0.4+0.3+1.0=1.700.4+0.3+1.0=1.70σ\sigma0.84550.8455
candidate gg1.0−0.25+0=0.751.0-0.25+0=0.75tanh⁡\tanh0.63510.6351
output oo−0.2+0.4+0.2=0.40-0.2+0.4+0.2=0.40σ\sigma0.59870.5987
Pre-activation and value of each gate (checked numerically in Node)

Forward, then the gradient back to the previous cell state

  1. 1Memory update: ct=f ct−1+i g=0.8455⋅0.8+0.6106⋅0.6351=0.6764+0.3878=1.0643c_t=f\,c_{t-1}+i\,g=0.8455\cdot0.8+0.6106\cdot0.6351=0.6764+0.3878=1.0643. The cell state is not bounded by 1: it is a sum, not a tanh.
  2. 2Output: ht=otanh⁡(ct)=0.5987⋅0.7873=0.4713h_t=o\tanh(c_t)=0.5987\cdot0.7873=0.4713.
  3. 3Suppose ∂L/∂ht=1\partial L/\partial h_t=1. Then ∂L/∂ct=o (1−tanh⁡2ct)=0.5987⋅(1−0.78732)=0.2276\partial L/\partial c_t=o\,(1-\tanh^2c_t)=0.5987\cdot(1-0.7873^2)=0.2276.
  4. 4Gate gradients: ∂L/∂f=0.2276⋅ct−1=0.1821\partial L/\partial f=0.2276\cdot c_{t-1}=0.1821, ∂L/∂i=0.2276⋅g=0.1446\partial L/\partial i=0.2276\cdot g=0.1446, ∂L/∂g=0.2276⋅i=0.1390\partial L/\partial g=0.2276\cdot i=0.1390, and ∂L/∂o=tanh⁡ct=0.7873\partial L/\partial o=\tanh c_t=0.7873. Each is then multiplied by its activation's derivative to reach the weights.
  5. 5Along the cell path, ∂L/∂ct−1+=0.2276⋅f=0.1924\partial L/\partial c_{t-1}\mathrel{+}=0.2276\cdot f=0.1924: the error crosses the step multiplied by f=0.8455f=0.8455 and nothing else.
python
1import torch, torch.nn as nn
2
3torch.manual_seed(0)
4N, L, I, H = 4, 7, 10, 20
5lstm = nn.LSTM(input_size=I, hidden_size=H, num_layers=1, batch_first=True)
6x = torch.randn(N, L, I) # (batch, time, features) because batch_first=True
7
8out, (h_n, c_n) = lstm(x) # h_0 and c_0 default to zeros
9print(out.shape, h_n.shape, c_n.shape) # (4, 7, 20) (1, 4, 20) (1, 4, 20)
10print(sum(p.numel() for p in lstm.parameters())) # 2560 = 4 * (H*I + H*H + 2*H)
11print(lstm.weight_ih_l0.shape) # (80, 10): rows [i | f | g | o], H each
12
13# Forget-gate bias of 1 (the second block of four; the two bias vectors are summed)
14with torch.no_grad():
15 lstm.bias_ih_l0[H:2 * H].fill_(1.0)
16 lstm.bias_hh_l0[H:2 * H].zero_()
17
18def step(x_t, h, c):
19 z = x_t @ lstm.weight_ih_l0.T + lstm.bias_ih_l0 + h @ lstm.weight_hh_l0.T + lstm.bias_hh_l0
20 i, f, g, o = z.chunk(4, dim=-1)
21 c = f.sigmoid() * c + i.sigmoid() * g.tanh()
22 return o.sigmoid() * c.tanh(), c
23
24h = c = torch.zeros(N, H)
25for t in range(L):
26 h, c = step(x[:, t], h, c)
27out, (h_n, c_n) = lstm(x)
28print(torch.allclose(h, h_n[0], atol=1e-5), torch.allclose(c, c_n[0], atol=1e-5)) # True True
29print(torch.allclose(out[:, -1], h_n[0])) # the last output is the final hidden state
30
31# The same step with the cell module, e.g. for custom loops or autoregressive decoding
32cell = nn.LSTMCell(I, H)
33h, c = cell(x[:, 0], (torch.zeros(N, H), torch.zeros(N, H)))
nn.LSTM shapes, gate packing, forget bias 1, and a hand-written step that matches the built-in layer

Variants

variantchangewhere
Original (1997)no forget gate: ct=ct−1+it⊙gtc_t=c_{t-1}+i_t\odot g_t; truncated gradientHochreiter & Schmidhuber
Forget gatect=ft⊙ct−1+it⊙gtc_t=f_t\odot c_{t-1}+i_t\odot g_tGers, Schmidhuber & Cummins 2000; the standard today
Peepholegates also read the cell: pi⊙ct−1p_i\odot c_{t-1}, pf⊙ct−1p_f\odot c_{t-1}, po⊙ctp_o\odot c_tGers & Schmidhuber 2000; not in nn.LSTM
Coupled gates (CIFG)ft=1−itf_t=1-i_t: write exactly as much as is forgottenGreff et al. 2017; the idea behind the GRU's update gate
Projection (LSTMP)ht=Whr(ot⊙tanh⁡ct)h_t=W_{hr}\big(o_t\odot\tanh c_t\big) with fewer rows than HHSak et al. 2014; proj_size in nn.LSTM
Common LSTM variants

Greff, Srivastava, Koutník, Steunebrink and Schmidhuber ("LSTM: A Search Space Odyssey", IEEE TNNLS 2017) compared eight variants on speech (TIMIT), handwriting (IAM Online) and polyphonic music (JSB Chorales) in about 5400 runs. None improved significantly on the standard cell with forget gate and peepholes, and the forget gate and the output activation (the tanh on ctc_t) were the most critical parts. Coupling the input and forget gates or removing peepholes did not hurt significantly, which is part of why the simpler GRU works as well as it does.

Use when

  • +Learns dependencies over hundreds of steps: the cell path scales the gradient only by ftf_t
  • +Gates make memory content-dependent: write, keep and read are decided per unit and per step
  • +Constant memory and compute per step at inference: well suited to streaming
  • +Mature, robust and fused in cuDNN; works well with modest tuning

Avoid when

  • −Sequential over time: step tt needs ht−1h_{t-1}, so training does not parallelise across the sequence like attention
  • −4×4\times the parameters and FLOPs of a vanilla RNN, 43×\tfrac43\times a GRU
  • −The state is a fixed-size vector: very long contexts must be compressed into HH numbers
  • −Still needs gradient clipping and careful dropout; the gate path still vanishes

Check yourself

A forget gate stays at f = 0.5 for 100 steps (zero bias, small weights). By what factor is the gradient along the cell path scaled between step 0 and step 100?

Check yourself

How many parameters does nn.LSTM(input_size=10, hidden_size=20) have?

Remember

  • ✓Two states: memory ct=ft⊙ct−1+it⊙gtc_t=f_t\odot c_{t-1}+i_t\odot g_t and output ht=ot⊙tanh⁡(ct)h_t=o_t\odot\tanh(c_t); input, forget and output gates are sigmoids of (xt,ht−1)(x_t,h_{t-1}).
  • ✓Along the cell path ∂ct/∂ct−1=diag⁡(ft)\partial c_t/\partial c_{t-1}=\operatorname{diag}(f_t): no weight matrix, no tanh⁡′\tanh'. With f≈1f\approx1 memory and gradient survive (the constant error carousel).
  • ✓Hochreiter & Schmidhuber (1997) had no forget gate; Gers et al. (2000) added it so cells can reset; peepholes are optional, and Greff et al. (2017) found no variant clearly better than the standard cell.
  • ✓Initialise the forget bias positive (1 is the usual choice); in PyTorch it is the slice [H:2H] of the packed i, f, g, o rows, and the two bias vectors add.
  • ✓Parameters 4(HI+H2+2H)4(HI+H^2+2H) per layer: four times a vanilla RNN, four thirds of a GRU.
← back to Deep Learning roadmap