Definition
Gated Linear Unit (GLU) is the layer : one linear projection (the value) multiplied elementwise by a sigmoid of a second linear projection (the gate). It was introduced by Dauphin, Fan, Auli and Grangier (2016) for gated convolutional language models.
An ordinary activation bends one pre-activation: . A GLU computes two pre-activations from the same input and lets one decide how much of the other gets through. Each output unit is a product of two linear functions of (one squashed into ), so the layer can express multiplicative interactions that a single linear map followed by a pointwise nonlinearity cannot.
The value path stays linear: there is no squashing function on . That is the point of the design, and it is what separates GLU from the older tanh-based gate.
Formula
- value: a plain linear projection, unbounded
- gate pre-activation, a second projection of the same input
- logistic sigmoid , so each gate is a soft switch in
- elementwise product: unit outputs
In code the two projections are usually fused into one matrix of width whose output is split in half. torch.nn.GLU(dim=-1) does only the split-and-gate step: for an input along dim it returns , halving that dimension.
Gradient
By the product rule, the gradient of one unit splits into a value path and a gate path:
- value-path gain: the gate itself, no derivative of a squashing function
- gate-path gain, at most
- column of each weight matrix
Dauphin et al. write it for a whole layer and contrast it with the gated tanh unit (GTU) used in WaveNet and PixelCNN (van den Oord et al., 2016):
- GLU's first term: the gradient flows through any open gate unscaled
- GTU's extra factor , which vanishes when the value saturates
So GLU gives the gradient a linear path through open gates, like the identity path in an LSTM cell or a residual connection, while the gate still provides a nonlinearity. The GTU has to pass the gradient through two saturating functions.
Properties
- Range
- unbounded (the value is linear); the gate is in
- Parameters
- weights and biases: twice a linear layer of the same output width
- Shape
nn.GLU(dim)maps size to alongdim- Not pointwise
- each output is a function of two pre-activations; there is no single curve to plot
- Smoothness
- ; the output is exactly 0 on the hyperplane , and near 0 wherever the gate is closed
- Gradient
- value path (no factor); gate path
- Family
- replace σ with another function: Bilinear (identity), ReGLU, GEGLU, SwiGLU
Intuition and history
Gating is old: LSTMs (Hochreiter & Schmidhuber, 1997) use sigmoid gates to decide what to write, keep and read. Dauphin et al. (arXiv, December 2016; ICML 2017) stripped the idea down to one multiplicative gate per unit inside a stack of causal 1-D convolutions, . Their non-recurrent model reached state-of-the-art results on WikiText-103 and competitive results on Google Billion Word, and in their comparisons GLU beat the GTU as well as ungated tanh and ReLU convolutions.
The gate is input-dependent: the same unit can pass its value for one input and block it for another. In the figure the output is the value map with a sigmoid-shaped "window" laid over it; move the gate weights and the window slides and rotates. A ReLU can only zero a unit along its own pre-activation's sign; a GLU unit decides using a different direction in input space.
Shazeer (2020) swapped the sigmoid for other functions inside the Transformer feed-forward layer. The resulting family is now standard in large language models:
| Variant | f | Formula | Notes |
|---|---|---|---|
| GLU | Dauphin et al. 2016, gated conv LM | ||
| Bilinear | identity | no nonlinearity except the product itself (Shazeer 2020) | |
| ReGLU | ReLU | hard gate: closed means exactly zero gradient | |
| GEGLU | GELU | T5 v1.1, Gemma; see GeGLU | |
| SwiGLU | Swish | PaLM, LLaMA, Mistral; see SwiGLU |
Worked example: one unit, forward and backward
Input , value weights , , gate weights , (the figure's defaults, probe at ). Take the upstream gradient .
| Quantity | Computation | Value |
|---|---|---|
| 0.7000 | ||
| −0.8000 | ||
| 0.3100 | ||
| 0.2170 | ||
| 0.3100 | ||
| 0.1497 | ||
| (0.3100, −0.1550) | ||
| (0.1497, −0.0749) | ||
| (0.1732, 0.2737) |
A GTU with the same pre-activations gives and value-path gain , already about a third below GLU's 0.3100. The gap grows with the value: at the GTU's gain is (a factor smaller) and at it is . GLU's value-path gain is regardless of .
Parameter count. A GLU layer from to holds weights, twice a plain linear layer. Fair comparisons therefore shrink the gated layer's width; the GLU-variant FFNs in Transformers use of the usual hidden size so three matrices cost what two did.
1import torch, torch.nn as nn, torch.nn.functional as F23glu = nn.GLU(dim=-1) # [A, B] -> A * sigmoid(B)4print(glu(torch.tensor([1.0, -2.0, 0.5, 3.0]))) # [ 0.6225, -1.9051]56class GLULayer(nn.Module):7 def __init__(self, d_in, d_out, gate=torch.sigmoid):8 super().__init__()9 self.proj = nn.Linear(d_in, 2 * d_out) # value and gate in one matmul10 self.gate = gate # sigmoid=GLU, F.relu=ReGLU, F.gelu=GEGLU, F.silu=SwiGLU11 def forward(self, x):12 a, g = self.proj(x).chunk(2, dim=-1)13 return a * self.gate(g)1415layer = GLULayer(2, 1)16with torch.no_grad():17 layer.proj.weight.copy_(torch.tensor([[0.8, 0.4], [-0.5, 1.0]]))18 layer.proj.bias.copy_(torch.tensor([0.1, 0.2]))19x = torch.tensor([[1.0, -0.5]], requires_grad=True)20y = layer(x); y.sum().backward()21print(y) # [[0.2170]]22print(layer.proj.weight.grad) # [[0.3100, -0.1550], [0.1497, -0.0749]]23print(x.grad) # [[0.1732, 0.2737]]Use when
- +Input-dependent gating with a linear gradient path through open gates.
- +Multiplicative feature interactions in a single layer.
- +Consistently better than ReLU/GELU feed-forward layers at equal parameters in Transformer language models (Shazeer 2020).
- +Trivial to implement: one wider matmul, a split, a pointwise product.
Avoid when
- −Twice the weights of a linear layer of the same width; width must be cut to compare fairly.
- −Three matrices in an FFN instead of two: more kernel launches and activation memory per token.
- −Unbounded output and an output that can be large where both and the gate are large.
- −No theory says why it works better; the evidence is empirical.
Check yourself
A GLU unit has value and an open gate . What is , and what would a GTU, , give?
Check yourself
nn.GLU(dim=-1) receives a tensor of shape (8, 128, 1024). What shape comes out?
Remember
- ✓: a linear value times a sigmoid gate (Dauphin et al., 2016).
- ✓Gradient: value path , gate path ; no factor, unlike the GTU.
- ✓Twice the weights of a linear layer;
nn.GLUsplits a -wide input into outputs. - ✓Gates are input-dependent and multiplicative: one layer can express products of features.
- ✓Family: Bilinear, ReGLU, GEGLU, SwiGLU; the FFN versions use of the hidden width to match parameters.
- ✓Worked example: , , , .