Gated recurrent unit (GRU)

A lighter gated RNN with no separate cell: an update gate interpolates h_t = (1−z_t)⊙n_t + z_t⊙h_{t−1}, and a reset gate decides how much of the past enters the candidate n_t. Three gate blocks instead of the LSTM's four, and in comparisons usually on par with it.

12 min read1140 wordscore0 viewsupdated 2026-10-12

Definition

Gated recurrent unit (GRU) is a recurrent cell with a single state hth_t and two sigmoid gates. The update gate ztz_t decides, per unit, how much of the old state to keep; the rest, 1−zt1-z_t, is replaced by a candidate ntn_t. The reset gate rtr_t decides how much of the old state is used when that candidate is computed. The new state is an interpolation, ht=(1−zt)⊙nt+zt⊙ht−1h_t=(1-z_t)\odot n_t+z_t\odot h_{t-1} (PyTorch's convention).

The GRU keeps the idea that makes the Long short-term memory (LSTM) work, an additive, gated path through time, and drops the rest. There is no separate cell state and no output gate. The input and forget gates are merged into one update gate, so whatever is written is exactly what is forgotten. The result has three weight blocks instead of four, about 25% fewer parameters and FLOPs, and the same ability to carry information across long gaps.

Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio introduced the unit in "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation" (EMNLP 2014), as the hidden unit of their encoder–decoder (see Encoder–decoder architecture). They described it as "a new type of hidden unit" motivated by the LSTM. The name gated recurrent unit was established by Chung, Gülçehre, Cho and Bengio ("Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling", NIPS 2014 Deep Learning Workshop), who compared it with the LSTM and the tanh RNN.

The equations (PyTorch convention)

rt=σ(Wirxt+bir+Whrht−1+bhr)reset gatezt=σ(Wizxt+biz+Whzht−1+bhz)update gatent=tanh⁡(Winxt+bin+rt⊙(Whnht−1+bhn))candidateht=(1−zt)⊙nt+zt⊙ht−1\begin{aligned} r_t&=\sigma(W_{ir}x_t+b_{ir}+W_{hr}h_{t-1}+b_{hr}) && \text{reset gate}\\ z_t&=\sigma(W_{iz}x_t+b_{iz}+W_{hz}h_{t-1}+b_{hz}) && \text{update gate}\\ n_t&=\tanh\big(W_{in}x_t+b_{in}+r_t\odot(W_{hn}h_{t-1}+b_{hn})\big) && \text{candidate}\\ h_t&=(1-z_t)\odot n_t+z_t\odot h_{t-1} \end{aligned}
zt∈(0,1)Hz_t\in(0,1)^H
update gate: the share of ht−1h_{t-1} that is kept. z=1z=1 copies the old state, z=0z=0 replaces it with ntn_t
rt∈(0,1)Hr_t\in(0,1)^H
reset gate: scales the recurrent contribution to the candidate. r=0r=0 makes ntn_t depend on xtx_t alone
nt∈(−1,1)Hn_t\in(-1,1)^H
candidate state, the "new content" (PyTorch calls it the new gate)
hth_t
the only state: it is both the memory and the output (no output gate, no cell)

Two conventions circulate, and they describe the same model. Cho et al. (2014), PyTorch and Keras let zz weight the old state, as above. Chung et al. (2014) and many tutorials write ht=(1−zt)⊙ht−1+zt⊙h~th_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t, where zz weights the new content. One is the other with z↔1−zz\leftrightarrow1-z, which the network absorbs by flipping the sign of the gate's weights and bias. It matters when you read a trained model's gates or port weights: in PyTorch, high zz means "remember".

A second, real difference is where the reset gate sits. Cho et al. reset the state before the matrix product, tanh⁡(Wxt+U(rt⊙ht−1))\tanh\big(Wx_t+U(r_t\odot h_{t-1})\big). PyTorch and cuDNN reset after it, tanh⁡(Winxt+bin+rt⊙(Whnht−1+bhn))\tanh\big(W_{in}x_t+b_{in}+r_t\odot(W_{hn}h_{t-1}+b_{hn})\big), so that Whnht−1W_{hn}h_{t-1} can be computed for all three gates in one fused product before rtr_t is known. Keras exposes both through reset_after (default True in TF 2). The two versions are not weight-compatible.

At a glance

State
one vector ht∈(−1,1)Hh_t\in(-1,1)^H; no separate cell, no output gate
Gates
reset rr, update zz (sigmoid) and the tanh candidate nn: three blocks of HH units
Parameters (PyTorch)
3(HI+H2+2H)3\big(HI+H^2+2H\big) per layer; nn.GRU(10, 20) has 3⋅20⋅32=19203\cdot20\cdot32=1920
vs LSTM
34\tfrac34 of the parameters and gate FLOPs (LSTM: 25602560 for the same sizes)
Packing order
weight_ih_l0 rows are (Wir ∣ Wiz ∣ Win)(W_{ir}\,|\,W_{iz}\,|\,W_{in}), shape 3H×I3H\times I
Gradient
∂ht/∂ht−1=diag⁡(zt)+\partial h_t/\partial h_{t-1}=\operatorname{diag}(z_t)+{}terms through ntn_t and ztz_t
Introduced
Cho et al., EMNLP 2014; named and benchmarked by Chung et al., 2014

Interpolation: a leaky integrator with a learned, input-dependent rate

With zz held constant, ht=(1−z)nt+z ht−1h_t=(1-z)n_t+z\,h_{t-1} is an exponential moving average of the candidates with time constant 1/(1−z)1/(1-z): the old state loses half its weight in ln⁡12/ln⁡z\ln\tfrac12/\ln z steps. That alone is a poor memory, because it averages everything. The gate makes the rate depend on the input and the state. When zt≈0z_t\approx0 the unit overwrites itself with ntn_t; when zt≈1z_t\approx1 it holds, almost untouched, however long that lasts. Because hth_t is a convex combination of nt∈(−1,1)n_t\in(-1,1) and ht−1h_{t-1}, the state stays in (−1,1)(-1,1) for free (the LSTM's cell state does not).

hₜ = (1 − zₜ)·nₜ + zₜ·hₜ₋₁, nₜ = tanh(1.2·xₜ + rₜ·1.5·hₜ₋₁), zₜ = σ(b_z − 10·cueₜ)xₜ▼▼▼▼zₜrₜ−1−0.500.51t=0t=15t=30t=45t=59━ hₜ ● candidate nₜ ┄ target (last cued value) ▼ write cue · dark strip = gate ≈ 1
t=32: h = (1 − 0.982)·0.464 + 0.982·−0.483 = −0.466 · target −0.624
MSE (t ≥ 4) 0.0083hold z = σ(b_z) = 0.982, half-life 38.2 steps∂h32/∂h15 = 0.739
r = 0.018
fig.One GRU unit with hand-set weights does sample-and-hold: 60 random inputs, four write cues (t = 4, 15, 33, 45), and the target is the cued value's clean candidate tanh(1.2·x), held until the next cue. The cue drives zₜ = σ(b_z − 10·cue). Defaults b_z = 4, b_r = −4: the gate holds with z = 0.982 (half-life 38.2 steps), opens to z = 0.0025 on a cue, and h tracks the target with MSE 0.0083. At t = 33, h = (1 − 0.0025)·0.770 + 0.0025·(−0.466) = 0.767 against a target of 0.775. The 1.8% written at each later step lets it drift to 0.638 by t = 44. Raise b_r to +4 (r = 0.982) and the old state leaks into each candidate: after the cue at t = 15 the unit holds +0.23 instead of −0.62, and MSE rises to 0.506. Switch 'cue drives z' off and the unit is a plain leaky average (MSE 0.485). b_z = 6 holds better (MSE 0.0016), b_z = 1 forgets within about 2 steps (MSE 0.290).

Gradient through time

∂ht∂ht−1=diag⁡(zt)+diag⁡(1−zt) ∂nt∂ht−1+diag⁡(ht−1−nt) ∂zt∂ht−1\frac{\partial h_t}{\partial h_{t-1}}=\operatorname{diag}(z_t)+\operatorname{diag}(1-z_t)\,\frac{\partial n_t}{\partial h_{t-1}}+\operatorname{diag}(h_{t-1}-n_t)\,\frac{\partial z_t}{\partial h_{t-1}}
diag⁡(zt)\operatorname{diag}(z_t)
the direct path: like the LSTM's diag⁡(ft)\operatorname{diag}(f_t), no weight matrix and no tanh⁡′\tanh'
∂nt∂ht−1\tfrac{\partial n_t}{\partial h_{t-1}}
diag⁡((1−nt2)⊙rt)Whn+…\operatorname{diag}\big((1-n_t^2)\odot r_t\big)W_{hn}+\dots: a vanilla-RNN-like term, damped by rtr_t and by 1−zt1-z_t
∂zt∂ht−1\tfrac{\partial z_t}{\partial h_{t-1}}
diag⁡(zt(1−zt))Whz\operatorname{diag}\big(z_t(1-z_t)\big)W_{hz}; small when the gate is saturated near 0 or 1

Over a span where the unit holds (z≈1z\approx1), the product of these Jacobians is close to ∏kzk\prod_k z_k, so the gradient survives the way it does along an LSTM's cell path. In the figure's default run, ∂h44/∂h33=0.822\partial h_{44}/\partial h_{33}=0.822, close to z11=0.98211=0.819z^{11}=0.982^{11}=0.819; the small extra comes from the path through ntn_t. The update gate therefore plays the role of the LSTM's forget gate, and the same advice applies: a positive bias on zz (in PyTorch's convention) starts the unit in remembering mode.

Worked example: one step of a one-unit GRU

Take xt=1x_t=1, ht−1=0.5h_{t-1}=0.5 and weights (wi,bi,wh,bh)(w_i,b_i,w_h,b_h): reset (0.5,0,−0.4,0)(0.5,0,-0.4,0), update (0.3,0,0.8,0)(0.3,0,0.8,0), candidate (1.0,0,0.7,0.1)(1.0,0,0.7,0.1).

Forward (checked numerically in Node)

  1. 1Reset gate: r=σ(0.5−0.2)=σ(0.3)=0.5744r=\sigma(0.5-0.2)=\sigma(0.3)=0.5744.
  2. 2Update gate: z=σ(0.3+0.4)=σ(0.7)=0.6682z=\sigma(0.3+0.4)=\sigma(0.7)=0.6682.
  3. 3Candidate, reset after the product: Whnh+bhn=0.35+0.1=0.45W_{hn}h+b_{hn}=0.35+0.1=0.45, so n=tanh⁡(1.0+0.5744⋅0.45)=tanh⁡(1.2585)=0.8506n=\tanh(1.0+0.5744\cdot0.45)=\tanh(1.2585)=0.8506.
  4. 4Interpolate: ht=(1−0.6682)⋅0.8506+0.6682⋅0.5=0.2823+0.3341=0.6163h_t=(1-0.6682)\cdot0.8506+0.6682\cdot0.5=0.2823+0.3341=0.6163.
conventioncandidatenew state h_t
PyTorch / cuDNN: zz keeps, reset after WhnhW_{hn}h0.85060.85060.61630.6163
Cho et al. 2014: zz keeps, reset before, tanh⁡(x+0.7 (rh)+0.1)\tanh(x+0.7\,(r h)+0.1)0.86200.86200.62010.6201
Chung et al. 2014 form: zz weights the candidate0.85060.85060.73430.7343
The same weights under the other conventions (checked numerically in Node)

The first two rows differ only slightly here, but they are different functions of the weights. The third row shows why the convention must be known before you read or transplant gate values: with identical weights, the swapped interpolation moves the state much further toward the candidate.

python
1import torch, torch.nn as nn
2
3torch.manual_seed(0)
4N, L, I, H = 4, 7, 10, 20
5gru = nn.GRU(input_size=I, hidden_size=H, batch_first=True)
6x = torch.randn(N, L, I)
7
8out, h_n = gru(x) # no cell state: just h_n
9print(out.shape, h_n.shape) # (4, 7, 20) (1, 4, 20)
10print(sum(p.numel() for p in gru.parameters())) # 1920 = 3 * (H*I + H*H + 2*H)
11print(gru.weight_ih_l0.shape) # (60, 10): rows [r | z | n]
12
13def step(x_t, h):
14 gi = x_t @ gru.weight_ih_l0.T + gru.bias_ih_l0 # input part of all three blocks
15 gh = h @ gru.weight_hh_l0.T + gru.bias_hh_l0 # recurrent part, one fused product
16 i_r, i_z, i_n = gi.chunk(3, dim=-1)
17 h_r, h_z, h_n_ = gh.chunk(3, dim=-1)
18 r = torch.sigmoid(i_r + h_r)
19 z = torch.sigmoid(i_z + h_z)
20 n = torch.tanh(i_n + r * h_n_) # reset applied AFTER the product
21 return (1 - z) * n + z * h # z keeps the old state
22
23h = torch.zeros(N, H)
24for t in range(L):
25 h = step(x[:, t], h)
26print(torch.allclose(h, h_n[0], atol=1e-5)) # True
27
28# Start in "remember" mode: positive update-gate bias (block [H:2H])
29with torch.no_grad():
30 gru.bias_hh_l0[H:2 * H].fill_(1.0)
31
32cell = nn.GRUCell(I, H) # one step at a time
33h = cell(x[:, 0], torch.zeros(N, H))
nn.GRU shapes, gate packing and a hand-written step that matches it

GRU or LSTM?

Chung et al. (2014) compared tanh RNNs, LSTMs and GRUs of roughly equal parameter count on four polyphonic-music datasets (Nottingham, JSB Chorales, MuseData, Piano-midi) and two raw-speech datasets from Ubisoft. Both gated units clearly beat the tanh RNN, especially on speech, but neither won consistently. Their conclusion was that the choice may depend heavily on the dataset and task. Jozefowicz, Zaremba and Sutskever (ICML 2015) searched over ten thousand architectures and found the GRU ahead of the LSTM on most of their tasks except language modelling, and that a forget-gate bias of 1 closed most of the gap.

There are also formal differences. Weiss, Goldberg and Yahav (ACL 2018) showed that with finite precision an LSTM can implement counting, for example recognising anbna^nb^n, and does so in practice, while a GRU cannot. The LSTM's cell is an unbounded sum; the GRU's state is a convex combination that stays in (−1,1)(-1,1). In practice the GRU is the cheaper default when parameters, memory or latency matter (on-device speech, small time-series models), and the LSTM is preferred when the task needs counting or very precise long-range storage. When it matters, try both: the difference is usually smaller than the effect of tuning.

aspectLSTMGRU
Statesctc_t and hth_thth_t only
Gatesinput, forget, output + candidatereset, update + candidate
Keep / writeindependent: ftf_t and iti_tcoupled: ztz_t and 1−zt1-z_t
Exposes the memorythrough ot⊙tanh⁡(ct)o_t\odot\tanh(c_t)always, in full
Parameters4(HI+H2+2H)4(HI+H^2+2H)3(HI+H2+2H)3(HI+H^2+2H)
I=256, H=512I=256,\ H=5121,576,9601,182,720
LSTM and GRU side by side (per layer, input size I, hidden size H)

Use when

  • +Same long-range ability as the LSTM in most benchmarks, with 34\tfrac34 of the parameters and FLOPs
  • +One state, bounded in (−1,1)(-1,1) by construction: no cell to explode, simpler to stack and inspect
  • +Fewer hyperparameters and gates to initialise; often converges at least as fast
  • +Fused cuDNN kernel in every major framework

Avoid when

  • −Keeping and writing are coupled: it cannot hold a value and add to it at the same time
  • −Cannot count in finite precision (Weiss et al., 2018); weaker for some formal-language tasks
  • −Exposes its whole state: no output gate to hide memory from the next layer
  • −Several incompatible conventions (z direction, reset placement, gate order)

Check yourself

In PyTorch's GRU, a unit has z_t = 0.99 for 50 consecutive steps. What is it doing?

Check yourself

How many parameters does nn.GRU(input_size=10, hidden_size=20) have?

Remember

  • ✓PyTorch GRU: rt,zt=σ(⋅)r_t,z_t=\sigma(\cdot), nt=tanh⁡(Winxt+bin+rt⊙(Whnht−1+bhn))n_t=\tanh(W_{in}x_t+b_{in}+r_t\odot(W_{hn}h_{t-1}+b_{hn})), ht=(1−zt)⊙nt+zt⊙ht−1h_t=(1-z_t)\odot n_t+z_t\odot h_{t-1}.
  • ✓The update gate both keeps (zz) and writes (1−z1-z); the reset gate controls how much of ht−1h_{t-1} shapes the new content.
  • ✓The direct path gives ∂ht/∂ht−1≈diag⁡(zt)\partial h_t/\partial h_{t-1}\approx\operatorname{diag}(z_t): like the LSTM's forget gate, zz near 1 preserves state and gradient.
  • ✓3(HI+H2+2H)3(HI+H^2+2H) parameters per layer, 34\tfrac34 of an LSTM; Cho et al. (2014) introduced it, Chung et al. (2014) found it on par with the LSTM.
  • ✓Conventions differ (z direction, reset before or after the product, gate order): verify one step by hand before porting weights.
← back to Deep Learning roadmap