# Long short-term memory (LSTM)

> A recurrent cell with a separate memory c_t, updated additively as c_t = f_t⊙c_{t−1} + i_t⊙g_t, and sigmoid gates that decide what to write, keep and read. Along the cell path the gradient is scaled only by the forget gate, so information and error survive hundreds of steps.

Source: https://www.mlfoundry.in/learn/long-short-term-memory-lstm  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 14 min read · updated 2026-10-12

**Long short-term memory (LSTM):** is a recurrent cell that keeps two states: a **cell state** $c_t$, the memory, and a **hidden state** $h_t$, the output. Three sigmoid gates decide how the memory changes at each step: the **input gate** $i_t$ (how much new content to write), the **forget gate** $f_t$ (how much old memory to keep) and the **output gate** $o_t$ (how much of the memory to expose). The memory is updated by addition, $c_t=f_t\odot c_{t-1}+i_t\odot g_t$, not by squashing it through a matrix and a nonlinearity at every step.

A vanilla RNN rewrites its whole state every step, $h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b)$. Backpropagating through $k$ steps multiplies $k$ Jacobians $\operatorname{diag}(\tanh')\,W_{hh}$, and that product shrinks or blows up exponentially in $k$ (see **Recurrent neural network (RNN)** and **Vanilla RNN**). In practice a tanh RNN trained by BPTT struggles to use information more than about 10 to 20 steps old. The LSTM was designed to remove that limit: give the network a path through time on which the error signal is multiplied by (close to) 1 at every step, and let learned gates decide when to use it.

Sepp Hochreiter analysed the vanishing gradient in his 1991 diploma thesis (supervised by Jürgen Schmidhuber), and Bengio, Simard and Frasconi (1994) showed the same difficulty independently. Hochreiter and Schmidhuber's "Long Short-Term Memory" (*Neural Computation* 9(8), 1997) introduced the memory cell with a self-connection of fixed weight 1.0, the **constant error carousel** (CEC), protected by multiplicative input and output gates. They reported learning tasks with minimal time lags of more than 1000 steps, far beyond the RNNs of the time. The 1997 cell had **no forget gate**. Gers, Schmidhuber and Cummins ("Learning to Forget", *Neural Computation* 2000) added it, because on long continual input streams without resets the cell state could grow without bound. Gers and Schmidhuber (2000) also added **peephole** connections from the cell to the gates, for precise timing. The form used today, with forget gate and trained by full backpropagation through time, is the one of Graves and Schmidhuber (2005).

## The gate equations

$$
\begin{aligned}
i_t&=\sigma(W_{ii}x_t+b_{ii}+W_{hi}h_{t-1}+b_{hi}) && \text{input gate}\\
f_t&=\sigma(W_{if}x_t+b_{if}+W_{hf}h_{t-1}+b_{hf}) && \text{forget gate}\\
g_t&=\tanh(W_{ig}x_t+b_{ig}+W_{hg}h_{t-1}+b_{hg}) && \text{candidate}\\
o_t&=\sigma(W_{io}x_t+b_{io}+W_{ho}h_{t-1}+b_{ho}) && \text{output gate}\\
c_t&=f_t\odot c_{t-1}+i_t\odot g_t\\
h_t&=o_t\odot\tanh(c_t)
\end{aligned}
$$

- $x_t\in\mathbb R^{I},\ h_t,c_t\in\mathbb R^{H}$: input, hidden state and cell state; $h_0=c_0=0$ unless given
- $\sigma$: logistic sigmoid, so every gate value is a per-unit fraction in $(0,1)$
- $g_t$: the candidate content, in $(-1,1)$; PyTorch's documentation calls it the "cell gate"
- $\odot$: element-wise product: each of the H memory units has its own gates
- $b_{i\cdot},\ b_{h\cdot}$: PyTorch keeps two bias vectors (for cuDNN compatibility); only their sum matters

All four pre-activations are affine in the same pair $(x_t,h_{t-1})$, so implementations compute them with **one** matrix product of width $4H$ and split the result. In PyTorch the rows of `weight_ih_l0` (shape $4H\times I$) and `weight_hh_l0` ($4H\times H$) are packed in the order $(W_{\cdot i}\,|\,W_{\cdot f}\,|\,W_{\cdot g}\,|\,W_{\cdot o})$: input, forget, cell (candidate), output. Keras uses the same order.

**At a glance**

- **States:** cell $c_t$ (memory, unbounded in principle) and hidden $h_t=o_t\odot\tanh(c_t)\in(-1,1)$
- **Gates:** input $i$, forget $f$, output $o$ (sigmoid), plus the tanh candidate $g$: four blocks of $H$ units
- **Parameters (PyTorch):** $4\big(HI+H^2+2H\big)$ per layer; `nn.LSTM(10, 20)` has $4(200+400+40)=2560$
- **vs other cells:** $4\times$ a vanilla RNN of the same $H$ ($640$ for $I=10,H=20$), $\tfrac43\times$ a GRU ($1920$)
- **Gradient along the cell:** $\partial c_t/\partial c_{t-1}=\operatorname{diag}(f_t)$ plus indirect terms through $h_{t-1}$
- **Common init:** forget-gate bias $+1$ (Jozefowicz et al., 2015; Keras `unit_forget_bias=True`)
- **Not in nn.LSTM:** peepholes, layer norm inside the cell; `proj_size` adds an LSTMP projection (Sak et al., 2014)

## The cell state is a conveyor belt

Read the update line by line. The forget gate scales the old memory, the input gate scales a new candidate, and the two are **added**. If $f_t=1$ and $i_t=0$ the memory passes through unchanged, however many steps that lasts. If $f_t=0$ and $i_t=1$ it is overwritten. Anything in between blends the two, separately for each of the $H$ units. The output gate does not touch the memory at all. It only decides what the rest of the network sees through $h_t$, so a cell can hold a value for a long time without letting it disturb the computation until it is needed.

> Figure: One LSTM unit with hand-set weights: the input gate opens only on the write cue at t = 2 (value 0.8, so c₂ ≈ 0.88), the output gate only on the read cue at the last step (t = 39), and in between come 36 random distractors. The forget gate is σ(b_f) at every step, so along the cell path ∂c₃₉/∂c₂ = σ(b_f)³⁷. At the default b_f = 1, σ = 0.731 and the product is 9.25e−6: the memory has decayed to distractor noise by the time it is read (h₃₉ = −0.012). Raise b_f to 5: σ = 0.993, the product is 0.780 and the read gives h₃₉ = 0.563. Switch the forget gate off (the 1997 cell, f ≡ 1): the product is exactly 1 and h₃₉ = 0.671, but the erase cue can no longer clear the memory. With the forget gate on, b_f = 5 and the erase cue, one step with f = 9.1e−4 wipes it (c₃₉ = 0.002).

## Why the gradient survives: the constant error carousel

Differentiate the cell update with respect to the previous cell state. One term comes straight from the product $f_t\odot c_{t-1}$. The others go through $h_{t-1}=o_{t-1}\odot\tanh(c_{t-1})$, which feeds every gate at step $t$:

$$
\frac{\partial c_t}{\partial c_{t-1}}=\underbrace{\operatorname{diag}(f_t)}_{\text{cell path}}+\underbrace{\Big[\operatorname{diag}(c_{t-1})\tfrac{\partial f_t}{\partial h_{t-1}}+\operatorname{diag}(g_t)\tfrac{\partial i_t}{\partial h_{t-1}}+\operatorname{diag}(i_t)\tfrac{\partial g_t}{\partial h_{t-1}}\Big]\tfrac{\partial h_{t-1}}{\partial c_{t-1}}}_{\text{through the gates}},\qquad \frac{\partial c_T}{\partial c_s}\Big|_{\text{cell path}}=\prod_{k=s+1}^{T}\operatorname{diag}(f_k)
$$

- $\operatorname{diag}(f_t)$: the direct term: no weight matrix and no $\tanh'$, just the forget gate
- $\tfrac{\partial h_{t-1}}{\partial c_{t-1}}$: $\operatorname{diag}\big(o_{t-1}\odot(1-\tanh^2 c_{t-1})\big)$; the bracket also contains $W_{h\cdot}$ matrices, so these terms behave like a vanilla RNN's Jacobian
- $\prod_k f_k$: how much of the error at step $T$ reaches step $s$ along the memory; close to 1 when the forget gates stay open

In the 1997 cell the forget gate did not exist, so the cell path had factor exactly 1 at every step: error entering the cell "rides the carousel" back through time without decay. Hochreiter and Schmidhuber also truncated the gradient wherever it left the cell, so only the CEC carried error across time. With a forget gate the factor is $f_k$, which the network learns. Keeping $f\approx1$ preserves both the memory and its gradient. Dropping $f$ toward 0 forgets on purpose. The vanishing gradient is not abolished (the gate path still vanishes, and $\prod f_k$ decays if the gates are not near 1), but the network can now **choose** to keep a long path open.

> **Additive updates are the whole trick:** The same idea, an identity-like path through a gated or plain sum, reappears across deep learning. **Highway networks** (Srivastava et al., 2015) applied LSTM-style gating across depth instead of time, ResNet's $x+F(x)$ is the ungated version, and the **GRU** merges the LSTM's input and forget gates into one update gate. In every case the gradient has a route that is not repeatedly squashed by a nonlinearity.

## Forget-gate bias: start by remembering

At initialisation the weights are small, so each gate sits near $\sigma(b)$. With a zero bias $f\approx0.5$, and the cell path halves the gradient every step: $0.5^{100}\approx7.9\times10^{-31}$ after 100 steps. The network then has to learn to remember before it can learn anything long-range. Gers et al. (2000) already started their forget gates open, and Jozefowicz, Zaremba and Sutskever (ICML 2015), after evaluating more than ten thousand recurrent architectures, recommended a forget bias of 1: in their experiments it closed the gap between the LSTM and the GRU. A bias of 1 gives $\sigma(1)=0.731$, which still decays fast over long lags ($0.731^{100}\approx2.5\times10^{-14}$). It only makes remembering easier to learn. Tasks with very long dependencies often use larger values, and the figure shows why: $\sigma(3)^{100}=0.0078$, $\sigma(5)^{100}=0.51$.

> **Two bias vectors in PyTorch:** `nn.LSTM` adds `bias_ih` and `bias_hh`, both initialised $\mathcal U(-1/\sqrt H,\,1/\sqrt H)$ like the weights. To get an effective forget bias of 1, set the slice `[H:2H]` of one of them to 1 and the same slice of the other to 0. Filling both with 1 gives 2. The forget gate is the **second** block of four; filling `[0:H]` sets the input gate instead.

## Worked example: one step of a one-unit LSTM

Take $I=H=1$, input $x_t=1$, previous states $h_{t-1}=0.5$, $c_{t-1}=0.8$, and weights $(w_x,\,w_h,\,b)$ per gate: $i:(0.5,-0.3,0.1)$, $f:(0.4,0.6,1.0)$, $g:(1.0,-0.5,0)$, $o:(-0.2,0.8,0.2)$. The forget bias of 1.0 follows the usual initialisation.

Pre-activation and value of each gate (checked numerically in Node)

| block | pre-activation | activation | value |
| --- | --- | --- | --- |
| input $i$ | $0.5-0.15+0.1=0.45$ | $\sigma$ | $0.6106$ |
| forget $f$ | $0.4+0.3+1.0=1.70$ | $\sigma$ | $0.8455$ |
| candidate $g$ | $1.0-0.25+0=0.75$ | $\tanh$ | $0.6351$ |
| output $o$ | $-0.2+0.4+0.2=0.40$ | $\sigma$ | $0.5987$ |

**Forward, then the gradient back to the previous cell state**

1. Memory update: $c_t=f\,c_{t-1}+i\,g=0.8455\cdot0.8+0.6106\cdot0.6351=0.6764+0.3878=1.0643$. The cell state is not bounded by 1: it is a sum, not a tanh.
2. Output: $h_t=o\tanh(c_t)=0.5987\cdot0.7873=0.4713$.
3. Suppose $\partial L/\partial h_t=1$. Then $\partial L/\partial c_t=o\,(1-\tanh^2c_t)=0.5987\cdot(1-0.7873^2)=0.2276$.
4. Gate gradients: $\partial L/\partial f=0.2276\cdot c_{t-1}=0.1821$, $\partial L/\partial i=0.2276\cdot g=0.1446$, $\partial L/\partial g=0.2276\cdot i=0.1390$, and $\partial L/\partial o=\tanh c_t=0.7873$. Each is then multiplied by its activation's derivative to reach the weights.
5. Along the cell path, $\partial L/\partial c_{t-1}\mathrel{+}=0.2276\cdot f=0.1924$: the error crosses the step multiplied by $f=0.8455$ and nothing else.

```python
import torch, torch.nn as nn

torch.manual_seed(0)
N, L, I, H = 4, 7, 10, 20
lstm = nn.LSTM(input_size=I, hidden_size=H, num_layers=1, batch_first=True)
x = torch.randn(N, L, I)                         # (batch, time, features) because batch_first=True

out, (h_n, c_n) = lstm(x)                        # h_0 and c_0 default to zeros
print(out.shape, h_n.shape, c_n.shape)           # (4, 7, 20) (1, 4, 20) (1, 4, 20)
print(sum(p.numel() for p in lstm.parameters())) # 2560 = 4 * (H*I + H*H + 2*H)
print(lstm.weight_ih_l0.shape)                   # (80, 10): rows [i | f | g | o], H each

# Forget-gate bias of 1 (the second block of four; the two bias vectors are summed)
with torch.no_grad():
    lstm.bias_ih_l0[H:2 * H].fill_(1.0)
    lstm.bias_hh_l0[H:2 * H].zero_()

def step(x_t, h, c):
    z = x_t @ lstm.weight_ih_l0.T + lstm.bias_ih_l0 + h @ lstm.weight_hh_l0.T + lstm.bias_hh_l0
    i, f, g, o = z.chunk(4, dim=-1)
    c = f.sigmoid() * c + i.sigmoid() * g.tanh()
    return o.sigmoid() * c.tanh(), c

h = c = torch.zeros(N, H)
for t in range(L):
    h, c = step(x[:, t], h, c)
out, (h_n, c_n) = lstm(x)
print(torch.allclose(h, h_n[0], atol=1e-5), torch.allclose(c, c_n[0], atol=1e-5))  # True True
print(torch.allclose(out[:, -1], h_n[0]))        # the last output is the final hidden state

# The same step with the cell module, e.g. for custom loops or autoregressive decoding
cell = nn.LSTMCell(I, H)
h, c = cell(x[:, 0], (torch.zeros(N, H), torch.zeros(N, H)))
```

nn.LSTM shapes, gate packing, forget bias 1, and a hand-written step that matches the built-in layer

## Variants

Common LSTM variants

| variant | change | where |
| --- | --- | --- |
| Original (1997) | no forget gate: $c_t=c_{t-1}+i_t\odot g_t$; truncated gradient | Hochreiter & Schmidhuber |
| Forget gate | $c_t=f_t\odot c_{t-1}+i_t\odot g_t$ | Gers, Schmidhuber & Cummins 2000; the standard today |
| Peephole | gates also read the cell: $p_i\odot c_{t-1}$, $p_f\odot c_{t-1}$, $p_o\odot c_t$ | Gers & Schmidhuber 2000; not in nn.LSTM |
| Coupled gates (CIFG) | $f_t=1-i_t$: write exactly as much as is forgotten | Greff et al. 2017; the idea behind the GRU's update gate |
| Projection (LSTMP) | $h_t=W_{hr}\big(o_t\odot\tanh c_t\big)$ with fewer rows than $H$ | Sak et al. 2014; `proj_size` in nn.LSTM |

Greff, Srivastava, Koutník, Steunebrink and Schmidhuber ("LSTM: A Search Space Odyssey", *IEEE TNNLS* 2017) compared eight variants on speech (TIMIT), handwriting (IAM Online) and polyphonic music (JSB Chorales) in about 5400 runs. None improved significantly on the standard cell with forget gate and peepholes, and the forget gate and the output activation (the tanh on $c_t$) were the most critical parts. Coupling the input and forget gates or removing peepholes did not hurt significantly, which is part of why the simpler **GRU** works as well as it does.

> **The cell state can still explode or be dropped wrongly:** $c_t$ is a running sum, so with $f\approx1$ and an input gate that keeps writing, it can grow large. $\tanh(c_t)$ then saturates and $\partial h_t/\partial c_t\to0$. That was Gers et al.'s reason for the forget gate, and gradient clipping (see **Gradient clipping**) is still standard for LSTMs. Ordinary dropout on $h_{t-1}$ with a fresh mask at every step destroys the memory. Use dropout between layers (`dropout=` in nn.LSTM), variational dropout with one mask per sequence, or DropConnect on the recurrent weights as in AWD-LSTM (see **DropConnect**).

> **Where LSTMs were used:** From about 2013 to 2017 the LSTM was the default sequence model. Graves, Mohamed and Hinton (2013) reported 17.7% phoneme error on TIMIT with deep bidirectional LSTMs. Sutskever, Vinyals and Le (2014) translated with stacked LSTMs (see **Sequence-to-sequence (Seq2Seq)**), and Google's GNMT (2016) used an 8-layer LSTM encoder and decoder. Language models such as AWD-LSTM followed. Transformers (see **Self-Attention**) displaced LSTMs because their training parallelises over time steps. LSTMs remain common for streaming, on-device and small time-series models, where a constant-size state per step is an advantage.

**Strengths**

- Learns dependencies over hundreds of steps: the cell path scales the gradient only by $f_t$
- Gates make memory content-dependent: write, keep and read are decided per unit and per step
- Constant memory and compute per step at inference: well suited to streaming
- Mature, robust and fused in cuDNN; works well with modest tuning

**Limitations**

- Sequential over time: step $t$ needs $h_{t-1}$, so training does not parallelise across the sequence like attention
- $4\times$ the parameters and FLOPs of a vanilla RNN, $\tfrac43\times$ a GRU
- The state is a fixed-size vector: very long contexts must be compressed into $H$ numbers
- Still needs gradient clipping and careful dropout; the gate path still vanishes

**Question:** A forget gate stays at f = 0.5 for 100 steps (zero bias, small weights). By what factor is the gradient along the cell path scaled between step 0 and step 100?

- A. About 7.9e−31: the memory path is useless at that range
- B. Exactly 1, because the cell path has no weight matrix
- C. 0.5, because the gate is applied once
- D. About 0.51

**Answer:** A. About 7.9e−31: the memory path is useless at that range — Along the cell path $\partial c_{100}/\partial c_0=\prod_k f_k=0.5^{100}\approx7.9\times10^{-31}$. The path has no weight matrix, but it still multiplies one forget-gate value per step. Factor 1 needs $f\equiv1$, as in the 1997 cell. 0.51 is what you get with $f=\sigma(5)=0.9933$, which is why forget gates are initialised with a positive bias.

**Question:** How many parameters does nn.LSTM(input_size=10, hidden_size=20) have?

- A. 2560
- B. 2400
- C. 1920
- D. 640

**Answer:** A. 2560 — Four gate blocks of $H=20$ rows, each with $I=10$ input weights, $H=20$ recurrent weights and two biases: $4\cdot20\cdot(10+20+2)=2560$. 2400 forgets the biases, 1920 is the GRU (three blocks), 640 a vanilla RNN.

**Key points**

- Two states: memory $c_t=f_t\odot c_{t-1}+i_t\odot g_t$ and output $h_t=o_t\odot\tanh(c_t)$; input, forget and output gates are sigmoids of $(x_t,h_{t-1})$.
- Along the cell path $\partial c_t/\partial c_{t-1}=\operatorname{diag}(f_t)$: no weight matrix, no $\tanh'$. With $f\approx1$ memory and gradient survive (the constant error carousel).
- Hochreiter & Schmidhuber (1997) had no forget gate; Gers et al. (2000) added it so cells can reset; peepholes are optional, and Greff et al. (2017) found no variant clearly better than the standard cell.
- Initialise the forget bias positive (1 is the usual choice); in PyTorch it is the slice [H:2H] of the packed i, f, g, o rows, and the two bias vectors add.
- Parameters $4(HI+H^2+2H)$ per layer: four times a vanilla RNN, four thirds of a GRU.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
