# Gated recurrent unit (GRU)

> A lighter gated RNN with no separate cell: an update gate interpolates h_t = (1−z_t)⊙n_t + z_t⊙h_{t−1}, and a reset gate decides how much of the past enters the candidate n_t. Three gate blocks instead of the LSTM's four, and in comparisons usually on par with it.

Source: https://www.mlfoundry.in/learn/gated-recurrent-unit-gru  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 12 min read · updated 2026-10-12

**Gated recurrent unit (GRU):** is a recurrent cell with a single state $h_t$ and two sigmoid gates. The **update gate** $z_t$ decides, per unit, how much of the old state to keep; the rest, $1-z_t$, is replaced by a **candidate** $n_t$. The **reset gate** $r_t$ decides how much of the old state is used when that candidate is computed. The new state is an interpolation, $h_t=(1-z_t)\odot n_t+z_t\odot h_{t-1}$ (PyTorch's convention).

The GRU keeps the idea that makes the **Long short-term memory (LSTM)** work, an additive, gated path through time, and drops the rest. There is no separate cell state and no output gate. The input and forget gates are merged into one update gate, so whatever is written is exactly what is forgotten. The result has three weight blocks instead of four, about 25% fewer parameters and FLOPs, and the same ability to carry information across long gaps.

Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio introduced the unit in "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation" (EMNLP 2014), as the hidden unit of their encoder–decoder (see **Encoder–decoder architecture**). They described it as "a new type of hidden unit" motivated by the LSTM. The name *gated recurrent unit* was established by Chung, Gülçehre, Cho and Bengio ("Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling", NIPS 2014 Deep Learning Workshop), who compared it with the LSTM and the tanh RNN.

## The equations (PyTorch convention)

$$
\begin{aligned}
r_t&=\sigma(W_{ir}x_t+b_{ir}+W_{hr}h_{t-1}+b_{hr}) && \text{reset gate}\\
z_t&=\sigma(W_{iz}x_t+b_{iz}+W_{hz}h_{t-1}+b_{hz}) && \text{update gate}\\
n_t&=\tanh\big(W_{in}x_t+b_{in}+r_t\odot(W_{hn}h_{t-1}+b_{hn})\big) && \text{candidate}\\
h_t&=(1-z_t)\odot n_t+z_t\odot h_{t-1}
\end{aligned}
$$

- $z_t\in(0,1)^H$: update gate: the share of $h_{t-1}$ that is **kept**. $z=1$ copies the old state, $z=0$ replaces it with $n_t$
- $r_t\in(0,1)^H$: reset gate: scales the recurrent contribution to the candidate. $r=0$ makes $n_t$ depend on $x_t$ alone
- $n_t\in(-1,1)^H$: candidate state, the "new content" (PyTorch calls it the new gate)
- $h_t$: the only state: it is both the memory and the output (no output gate, no cell)

Two conventions circulate, and they describe the same model. Cho et al. (2014), PyTorch and Keras let $z$ weight the **old** state, as above. Chung et al. (2014) and many tutorials write $h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t$, where $z$ weights the **new** content. One is the other with $z\leftrightarrow1-z$, which the network absorbs by flipping the sign of the gate's weights and bias. It matters when you read a trained model's gates or port weights: in PyTorch, high $z$ means "remember".

A second, real difference is **where the reset gate sits**. Cho et al. reset the state before the matrix product, $\tanh\big(Wx_t+U(r_t\odot h_{t-1})\big)$. PyTorch and cuDNN reset after it, $\tanh\big(W_{in}x_t+b_{in}+r_t\odot(W_{hn}h_{t-1}+b_{hn})\big)$, so that $W_{hn}h_{t-1}$ can be computed for all three gates in one fused product before $r_t$ is known. Keras exposes both through `reset_after` (default `True` in TF 2). The two versions are not weight-compatible.

**At a glance**

- **State:** one vector $h_t\in(-1,1)^H$; no separate cell, no output gate
- **Gates:** reset $r$, update $z$ (sigmoid) and the tanh candidate $n$: three blocks of $H$ units
- **Parameters (PyTorch):** $3\big(HI+H^2+2H\big)$ per layer; `nn.GRU(10, 20)` has $3\cdot20\cdot32=1920$
- **vs LSTM:** $\tfrac34$ of the parameters and gate FLOPs (LSTM: $2560$ for the same sizes)
- **Packing order:** `weight_ih_l0` rows are $(W_{ir}\,|\,W_{iz}\,|\,W_{in})$, shape $3H\times I$
- **Gradient:** $\partial h_t/\partial h_{t-1}=\operatorname{diag}(z_t)+{}$terms through $n_t$ and $z_t$
- **Introduced:** Cho et al., EMNLP 2014; named and benchmarked by Chung et al., 2014

## Interpolation: a leaky integrator with a learned, input-dependent rate

With $z$ held constant, $h_t=(1-z)n_t+z\,h_{t-1}$ is an exponential moving average of the candidates with time constant $1/(1-z)$: the old state loses half its weight in $\ln\tfrac12/\ln z$ steps. That alone is a poor memory, because it averages everything. The gate makes the rate depend on the input and the state. When $z_t\approx0$ the unit **overwrites** itself with $n_t$; when $z_t\approx1$ it **holds**, almost untouched, however long that lasts. Because $h_t$ is a convex combination of $n_t\in(-1,1)$ and $h_{t-1}$, the state stays in $(-1,1)$ for free (the LSTM's cell state does not).

> Figure: One GRU unit with hand-set weights does sample-and-hold: 60 random inputs, four write cues (t = 4, 15, 33, 45), and the target is the cued value's clean candidate tanh(1.2·x), held until the next cue. The cue drives zₜ = σ(b_z − 10·cue). Defaults b_z = 4, b_r = −4: the gate holds with z = 0.982 (half-life 38.2 steps), opens to z = 0.0025 on a cue, and h tracks the target with MSE 0.0083. At t = 33, h = (1 − 0.0025)·0.770 + 0.0025·(−0.466) = 0.767 against a target of 0.775. The 1.8% written at each later step lets it drift to 0.638 by t = 44. Raise b_r to +4 (r = 0.982) and the old state leaks into each candidate: after the cue at t = 15 the unit holds +0.23 instead of −0.62, and MSE rises to 0.506. Switch 'cue drives z' off and the unit is a plain leaky average (MSE 0.485). b_z = 6 holds better (MSE 0.0016), b_z = 1 forgets within about 2 steps (MSE 0.290).

> **One gate, two jobs:** In an LSTM, keeping ($f$) and writing ($i$) are independent: a cell can keep everything **and** add something. In a GRU the shares are tied, $z$ kept plus $1-z$ written, summing to 1. That is the coupled input–forget gate that Greff et al. (2017) found does not hurt the LSTM, and it is why the figure's held value drifts: holding with $z=0.982$ means writing 1.8% of every distractor. Pushing $z$ closer to 1 (larger $b_z$) holds better, but then writing needs a stronger signal to open the gate.

## Gradient through time

$$
\frac{\partial h_t}{\partial h_{t-1}}=\operatorname{diag}(z_t)+\operatorname{diag}(1-z_t)\,\frac{\partial n_t}{\partial h_{t-1}}+\operatorname{diag}(h_{t-1}-n_t)\,\frac{\partial z_t}{\partial h_{t-1}}
$$

- $\operatorname{diag}(z_t)$: the direct path: like the LSTM's $\operatorname{diag}(f_t)$, no weight matrix and no $\tanh'$
- $\tfrac{\partial n_t}{\partial h_{t-1}}$: $\operatorname{diag}\big((1-n_t^2)\odot r_t\big)W_{hn}+\dots$: a vanilla-RNN-like term, damped by $r_t$ and by $1-z_t$
- $\tfrac{\partial z_t}{\partial h_{t-1}}$: $\operatorname{diag}\big(z_t(1-z_t)\big)W_{hz}$; small when the gate is saturated near 0 or 1

Over a span where the unit holds ($z\approx1$), the product of these Jacobians is close to $\prod_k z_k$, so the gradient survives the way it does along an LSTM's cell path. In the figure's default run, $\partial h_{44}/\partial h_{33}=0.822$, close to $z^{11}=0.982^{11}=0.819$; the small extra comes from the path through $n_t$. The update gate therefore plays the role of the LSTM's forget gate, and the same advice applies: a positive bias on $z$ (in PyTorch's convention) starts the unit in remembering mode.

## Worked example: one step of a one-unit GRU

Take $x_t=1$, $h_{t-1}=0.5$ and weights $(w_i,b_i,w_h,b_h)$: reset $(0.5,0,-0.4,0)$, update $(0.3,0,0.8,0)$, candidate $(1.0,0,0.7,0.1)$.

**Forward (checked numerically in Node)**

1. Reset gate: $r=\sigma(0.5-0.2)=\sigma(0.3)=0.5744$.
2. Update gate: $z=\sigma(0.3+0.4)=\sigma(0.7)=0.6682$.
3. Candidate, reset after the product: $W_{hn}h+b_{hn}=0.35+0.1=0.45$, so $n=\tanh(1.0+0.5744\cdot0.45)=\tanh(1.2585)=0.8506$.
4. Interpolate: $h_t=(1-0.6682)\cdot0.8506+0.6682\cdot0.5=0.2823+0.3341=0.6163$.

The same weights under the other conventions (checked numerically in Node)

| convention | candidate | new state h_t |
| --- | --- | --- |
| PyTorch / cuDNN: $z$ keeps, reset after $W_{hn}h$ | $0.8506$ | $0.6163$ |
| Cho et al. 2014: $z$ keeps, reset before, $\tanh(x+0.7\,(r h)+0.1)$ | $0.8620$ | $0.6201$ |
| Chung et al. 2014 form: $z$ weights the candidate | $0.8506$ | $0.7343$ |

The first two rows differ only slightly here, but they are different functions of the weights. The third row shows why the convention must be known before you read or transplant gate values: with identical weights, the swapped interpolation moves the state much further toward the candidate.

```python
import torch, torch.nn as nn

torch.manual_seed(0)
N, L, I, H = 4, 7, 10, 20
gru = nn.GRU(input_size=I, hidden_size=H, batch_first=True)
x = torch.randn(N, L, I)

out, h_n = gru(x)                                # no cell state: just h_n
print(out.shape, h_n.shape)                      # (4, 7, 20) (1, 4, 20)
print(sum(p.numel() for p in gru.parameters()))  # 1920 = 3 * (H*I + H*H + 2*H)
print(gru.weight_ih_l0.shape)                    # (60, 10): rows [r | z | n]

def step(x_t, h):
    gi = x_t @ gru.weight_ih_l0.T + gru.bias_ih_l0   # input part of all three blocks
    gh = h @ gru.weight_hh_l0.T + gru.bias_hh_l0     # recurrent part, one fused product
    i_r, i_z, i_n = gi.chunk(3, dim=-1)
    h_r, h_z, h_n_ = gh.chunk(3, dim=-1)
    r = torch.sigmoid(i_r + h_r)
    z = torch.sigmoid(i_z + h_z)
    n = torch.tanh(i_n + r * h_n_)                   # reset applied AFTER the product
    return (1 - z) * n + z * h                       # z keeps the old state

h = torch.zeros(N, H)
for t in range(L):
    h = step(x[:, t], h)
print(torch.allclose(h, h_n[0], atol=1e-5))      # True

# Start in "remember" mode: positive update-gate bias (block [H:2H])
with torch.no_grad():
    gru.bias_hh_l0[H:2 * H].fill_(1.0)

cell = nn.GRUCell(I, H)                          # one step at a time
h = cell(x[:, 0], torch.zeros(N, H))
```

nn.GRU shapes, gate packing and a hand-written step that matches it

## GRU or LSTM?

Chung et al. (2014) compared tanh RNNs, LSTMs and GRUs of roughly equal parameter count on four polyphonic-music datasets (Nottingham, JSB Chorales, MuseData, Piano-midi) and two raw-speech datasets from Ubisoft. Both gated units clearly beat the tanh RNN, especially on speech, but neither won consistently. Their conclusion was that the choice may depend heavily on the dataset and task. Jozefowicz, Zaremba and Sutskever (ICML 2015) searched over ten thousand architectures and found the GRU ahead of the LSTM on most of their tasks except language modelling, and that a forget-gate bias of 1 closed most of the gap.

There are also formal differences. Weiss, Goldberg and Yahav (ACL 2018) showed that with finite precision an LSTM can implement counting, for example recognising $a^nb^n$, and does so in practice, while a GRU cannot. The LSTM's cell is an unbounded sum; the GRU's state is a convex combination that stays in $(-1,1)$. In practice the GRU is the cheaper default when parameters, memory or latency matter (on-device speech, small time-series models), and the LSTM is preferred when the task needs counting or very precise long-range storage. When it matters, try both: the difference is usually smaller than the effect of tuning.

LSTM and GRU side by side (per layer, input size I, hidden size H)

| aspect | LSTM | GRU |
| --- | --- | --- |
| States | $c_t$ and $h_t$ | $h_t$ only |
| Gates | input, forget, output + candidate | reset, update + candidate |
| Keep / write | independent: $f_t$ and $i_t$ | coupled: $z_t$ and $1-z_t$ |
| Exposes the memory | through $o_t\odot\tanh(c_t)$ | always, in full |
| Parameters | $4(HI+H^2+2H)$ | $3(HI+H^2+2H)$ |
| $I=256,\ H=512$ | 1,576,960 | 1,182,720 |

> **Porting weights between frameworks:** A GRU checkpoint carries three conventions: the gate order inside the packed matrices (PyTorch $r,z,n$; Keras $z,r,h$), whether $z$ weights the old or the new state, and whether reset is applied before or after the recurrent product. Getting any one wrong gives a model that runs and produces plausible-looking garbage. Check by running one step by hand against the framework, as in the code above.

> **Where GRUs are used:** The GRU was the hidden unit of the first neural machine translation encoder–decoders (Cho et al., 2014) and of Bahdanau et al.'s attention model (see **Bahdanau attention**). It remains popular in speech front ends, recommender session models (GRU4Rec, Hidasi et al., 2016) and small forecasting models, where its smaller state and parameter count help. Like the LSTM it is sequential over time; see **Self-Attention** for the parallel alternative.

**Strengths**

- Same long-range ability as the LSTM in most benchmarks, with $\tfrac34$ of the parameters and FLOPs
- One state, bounded in $(-1,1)$ by construction: no cell to explode, simpler to stack and inspect
- Fewer hyperparameters and gates to initialise; often converges at least as fast
- Fused cuDNN kernel in every major framework

**Limitations**

- Keeping and writing are coupled: it cannot hold a value and add to it at the same time
- Cannot count in finite precision (Weiss et al., 2018); weaker for some formal-language tasks
- Exposes its whole state: no output gate to hide memory from the next layer
- Several incompatible conventions (z direction, reset placement, gate order)

**Question:** In PyTorch's GRU, a unit has z_t = 0.99 for 50 consecutive steps. What is it doing?

- A. Holding its state: each step keeps 99% of h and writes 1% of the candidate
- B. Overwriting its state with the candidate at every step
- C. Ignoring the past when computing the candidate
- D. Hiding its state from the next layer

**Answer:** A. Holding its state: each step keeps 99% of h and writes 1% of the candidate — $h_t=(1-z_t)n_t+z_t h_{t-1}$, so $z=0.99$ keeps 99% of the old state per step: after 50 steps $0.99^{50}=0.61$ of it is still there, and the direct-path gradient is about the same. Overwriting would be $z\approx0$. Ignoring the past in the candidate is the reset gate's job ($r\approx0$), and the GRU has no output gate. In Chung et al.'s convention the same behaviour shows up as $z=0.01$.

**Question:** How many parameters does nn.GRU(input_size=10, hidden_size=20) have?

- A. 1920
- B. 2560
- C. 1800
- D. 1280

**Answer:** A. 1920 — Three gate blocks of 20 rows, each with 10 input weights, 20 recurrent weights and two biases: $3\cdot20\cdot(10+20+2)=1920$. 2560 is the LSTM with four blocks, and 1800 forgets the biases.

**Key points**

- PyTorch GRU: $r_t,z_t=\sigma(\cdot)$, $n_t=\tanh(W_{in}x_t+b_{in}+r_t\odot(W_{hn}h_{t-1}+b_{hn}))$, $h_t=(1-z_t)\odot n_t+z_t\odot h_{t-1}$.
- The update gate both keeps ($z$) and writes ($1-z$); the reset gate controls how much of $h_{t-1}$ shapes the new content.
- The direct path gives $\partial h_t/\partial h_{t-1}\approx\operatorname{diag}(z_t)$: like the LSTM's forget gate, $z$ near 1 preserves state and gradient.
- $3(HI+H^2+2H)$ parameters per layer, $\tfrac34$ of an LSTM; Cho et al. (2014) introduced it, Chung et al. (2014) found it on par with the LSTM.
- Conventions differ (z direction, reset before or after the product, gate order): verify one step by hand before porting weights.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
