Definition
Gated recurrent unit (GRU) is a recurrent cell with a single state and two sigmoid gates. The update gate decides, per unit, how much of the old state to keep; the rest, , is replaced by a candidate . The reset gate decides how much of the old state is used when that candidate is computed. The new state is an interpolation, (PyTorch's convention).
The GRU keeps the idea that makes the Long short-term memory (LSTM) work, an additive, gated path through time, and drops the rest. There is no separate cell state and no output gate. The input and forget gates are merged into one update gate, so whatever is written is exactly what is forgotten. The result has three weight blocks instead of four, about 25% fewer parameters and FLOPs, and the same ability to carry information across long gaps.
Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio introduced the unit in "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation" (EMNLP 2014), as the hidden unit of their encoder–decoder (see Encoder–decoder architecture). They described it as "a new type of hidden unit" motivated by the LSTM. The name gated recurrent unit was established by Chung, Gülçehre, Cho and Bengio ("Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling", NIPS 2014 Deep Learning Workshop), who compared it with the LSTM and the tanh RNN.
The equations (PyTorch convention)
- update gate: the share of that is kept. copies the old state, replaces it with
- reset gate: scales the recurrent contribution to the candidate. makes depend on alone
- candidate state, the "new content" (PyTorch calls it the new gate)
- the only state: it is both the memory and the output (no output gate, no cell)
Two conventions circulate, and they describe the same model. Cho et al. (2014), PyTorch and Keras let weight the old state, as above. Chung et al. (2014) and many tutorials write , where weights the new content. One is the other with , which the network absorbs by flipping the sign of the gate's weights and bias. It matters when you read a trained model's gates or port weights: in PyTorch, high means "remember".
A second, real difference is where the reset gate sits. Cho et al. reset the state before the matrix product, . PyTorch and cuDNN reset after it, , so that can be computed for all three gates in one fused product before is known. Keras exposes both through reset_after (default True in TF 2). The two versions are not weight-compatible.
At a glance
- State
- one vector ; no separate cell, no output gate
- Gates
- reset , update (sigmoid) and the tanh candidate : three blocks of units
- Parameters (PyTorch)
- per layer;
nn.GRU(10, 20)has - vs LSTM
- of the parameters and gate FLOPs (LSTM: for the same sizes)
- Packing order
weight_ih_l0rows are , shape- Gradient
- terms through and
- Introduced
- Cho et al., EMNLP 2014; named and benchmarked by Chung et al., 2014
Interpolation: a leaky integrator with a learned, input-dependent rate
With held constant, is an exponential moving average of the candidates with time constant : the old state loses half its weight in steps. That alone is a poor memory, because it averages everything. The gate makes the rate depend on the input and the state. When the unit overwrites itself with ; when it holds, almost untouched, however long that lasts. Because is a convex combination of and , the state stays in for free (the LSTM's cell state does not).
Gradient through time
- the direct path: like the LSTM's , no weight matrix and no
- : a vanilla-RNN-like term, damped by and by
- ; small when the gate is saturated near 0 or 1
Over a span where the unit holds (), the product of these Jacobians is close to , so the gradient survives the way it does along an LSTM's cell path. In the figure's default run, , close to ; the small extra comes from the path through . The update gate therefore plays the role of the LSTM's forget gate, and the same advice applies: a positive bias on (in PyTorch's convention) starts the unit in remembering mode.
Worked example: one step of a one-unit GRU
Take , and weights : reset , update , candidate .
Forward (checked numerically in Node)
- 1Reset gate: .
- 2Update gate: .
- 3Candidate, reset after the product: , so .
- 4Interpolate: .
| convention | candidate | new state h_t |
|---|---|---|
| PyTorch / cuDNN: keeps, reset after | ||
| Cho et al. 2014: keeps, reset before, | ||
| Chung et al. 2014 form: weights the candidate |
The first two rows differ only slightly here, but they are different functions of the weights. The third row shows why the convention must be known before you read or transplant gate values: with identical weights, the swapped interpolation moves the state much further toward the candidate.
1import torch, torch.nn as nn23torch.manual_seed(0)4N, L, I, H = 4, 7, 10, 205gru = nn.GRU(input_size=I, hidden_size=H, batch_first=True)6x = torch.randn(N, L, I)78out, h_n = gru(x) # no cell state: just h_n9print(out.shape, h_n.shape) # (4, 7, 20) (1, 4, 20)10print(sum(p.numel() for p in gru.parameters())) # 1920 = 3 * (H*I + H*H + 2*H)11print(gru.weight_ih_l0.shape) # (60, 10): rows [r | z | n]1213def step(x_t, h):14 gi = x_t @ gru.weight_ih_l0.T + gru.bias_ih_l0 # input part of all three blocks15 gh = h @ gru.weight_hh_l0.T + gru.bias_hh_l0 # recurrent part, one fused product16 i_r, i_z, i_n = gi.chunk(3, dim=-1)17 h_r, h_z, h_n_ = gh.chunk(3, dim=-1)18 r = torch.sigmoid(i_r + h_r)19 z = torch.sigmoid(i_z + h_z)20 n = torch.tanh(i_n + r * h_n_) # reset applied AFTER the product21 return (1 - z) * n + z * h # z keeps the old state2223h = torch.zeros(N, H)24for t in range(L):25 h = step(x[:, t], h)26print(torch.allclose(h, h_n[0], atol=1e-5)) # True2728# Start in "remember" mode: positive update-gate bias (block [H:2H])29with torch.no_grad():30 gru.bias_hh_l0[H:2 * H].fill_(1.0)3132cell = nn.GRUCell(I, H) # one step at a time33h = cell(x[:, 0], torch.zeros(N, H))GRU or LSTM?
Chung et al. (2014) compared tanh RNNs, LSTMs and GRUs of roughly equal parameter count on four polyphonic-music datasets (Nottingham, JSB Chorales, MuseData, Piano-midi) and two raw-speech datasets from Ubisoft. Both gated units clearly beat the tanh RNN, especially on speech, but neither won consistently. Their conclusion was that the choice may depend heavily on the dataset and task. Jozefowicz, Zaremba and Sutskever (ICML 2015) searched over ten thousand architectures and found the GRU ahead of the LSTM on most of their tasks except language modelling, and that a forget-gate bias of 1 closed most of the gap.
There are also formal differences. Weiss, Goldberg and Yahav (ACL 2018) showed that with finite precision an LSTM can implement counting, for example recognising , and does so in practice, while a GRU cannot. The LSTM's cell is an unbounded sum; the GRU's state is a convex combination that stays in . In practice the GRU is the cheaper default when parameters, memory or latency matter (on-device speech, small time-series models), and the LSTM is preferred when the task needs counting or very precise long-range storage. When it matters, try both: the difference is usually smaller than the effect of tuning.
| aspect | LSTM | GRU |
|---|---|---|
| States | and | only |
| Gates | input, forget, output + candidate | reset, update + candidate |
| Keep / write | independent: and | coupled: and |
| Exposes the memory | through | always, in full |
| Parameters | ||
| 1,576,960 | 1,182,720 |
Use when
- +Same long-range ability as the LSTM in most benchmarks, with of the parameters and FLOPs
- +One state, bounded in by construction: no cell to explode, simpler to stack and inspect
- +Fewer hyperparameters and gates to initialise; often converges at least as fast
- +Fused cuDNN kernel in every major framework
Avoid when
- −Keeping and writing are coupled: it cannot hold a value and add to it at the same time
- −Cannot count in finite precision (Weiss et al., 2018); weaker for some formal-language tasks
- −Exposes its whole state: no output gate to hide memory from the next layer
- −Several incompatible conventions (z direction, reset placement, gate order)
Check yourself
In PyTorch's GRU, a unit has z_t = 0.99 for 50 consecutive steps. What is it doing?
Check yourself
How many parameters does nn.GRU(input_size=10, hidden_size=20) have?
Remember
- ✓PyTorch GRU: , , .
- ✓The update gate both keeps () and writes (); the reset gate controls how much of shapes the new content.
- ✓The direct path gives : like the LSTM's forget gate, near 1 preserves state and gradient.
- ✓ parameters per layer, of an LSTM; Cho et al. (2014) introduced it, Chung et al. (2014) found it on par with the LSTM.
- ✓Conventions differ (z direction, reset before or after the product, gate order): verify one step by hand before porting weights.