GLU (Gated Linear Unit)

Two linear projections, one passed through a sigmoid and multiplied into the other: an input-dependent gate with a linear path for the gradient.

9 min read756 wordsadvanced0 viewsupdated 2026-10-11

Definition

Gated Linear Unit (GLU) is the layer GLU⁡(x)=(xW+b)⊙σ(xV+c)\operatorname{GLU}(x)=(xW+b)\odot\sigma(xV+c): one linear projection (the value) multiplied elementwise by a sigmoid of a second linear projection (the gate). It was introduced by Dauphin, Fan, Auli and Grangier (2016) for gated convolutional language models.

An ordinary activation bends one pre-activation: y=φ(z)y=\varphi(z). A GLU computes two pre-activations from the same input and lets one decide how much of the other gets through. Each output unit is a product of two linear functions of xx (one squashed into (0,1)(0,1)), so the layer can express multiplicative interactions that a single linear map followed by a pointwise nonlinearity cannot.

The value path stays linear: there is no squashing function on xW+bxW+b. That is the point of the design, and it is what separates GLU from the older tanh-based gate.

Formula

GLU⁡(x)=(xW+b)⏟a (value)  ⊙  σ(xV+c⏟g (gate)),W,V∈Rdin×dout\operatorname{GLU}(x)=\underbrace{(xW+b)}_{a\ \text{(value)}}\;\odot\;\sigma\big(\underbrace{xV+c}_{g\ \text{(gate)}}\big),\qquad W,V\in\mathbb{R}^{d_{in}\times d_{out}}
a=xW+ba=xW+b
value: a plain linear projection, unbounded
g=xV+cg=xV+c
gate pre-activation, a second projection of the same input
σ\sigma
logistic sigmoid 1/(1+e−g)1/(1+e^{-g}), so each gate is a soft switch in (0,1)(0,1)
⊙\odot
elementwise product: unit kk outputs ak σ(gk)a_k\,\sigma(g_k)

In code the two projections are usually fused into one matrix of width 2dout2d_{out} whose output is split in half. torch.nn.GLU(dim=-1) does only the split-and-gate step: for an input [A, B][A,\,B] along dim it returns A⊙σ(B)A\odot\sigma(B), halving that dimension.

Gradient

By the product rule, the gradient of one unit splits into a value path and a gate path:

∂y∂a=σ(g),∂y∂g=a σ(g)(1−σ(g)),∂y∂x=σ(g) W:,k+a σ′(g) V:,k\frac{\partial y}{\partial a}=\sigma(g),\qquad \frac{\partial y}{\partial g}=a\,\sigma(g)\big(1-\sigma(g)\big),\qquad \frac{\partial y}{\partial x}=\sigma(g)\,W_{:,k}+a\,\sigma'(g)\,V_{:,k}
σ(g)\sigma(g)
value-path gain: the gate itself, no derivative of a squashing function
a σ′(g)a\,\sigma'(g)
gate-path gain, at most ∣a∣/4|a|/4
W:,k,V:,kW_{:,k},V_{:,k}
column kk of each weight matrix

Dauphin et al. write it for a whole layer and contrast it with the gated tanh unit (GTU) tanh⁡(X)⊙σ(X)\tanh(X)\odot\sigma(X) used in WaveNet and PixelCNN (van den Oord et al., 2016):

∇[X⊗σ(X)]=∇X⊗σ(X)+X⊗σ′(X)∇Xvs∇[tanh⁡(X)⊗σ(X)]=tanh⁡′(X)∇X⊗σ(X)+σ′(X)∇X⊗tanh⁡(X)\nabla[X\otimes\sigma(X)]=\nabla X\otimes\sigma(X)+X\otimes\sigma'(X)\nabla X\qquad\text{vs}\qquad \nabla[\tanh(X)\otimes\sigma(X)]=\tanh'(X)\nabla X\otimes\sigma(X)+\sigma'(X)\nabla X\otimes\tanh(X)
∇X⊗σ(X)\nabla X\otimes\sigma(X)
GLU's first term: the gradient flows through any open gate unscaled
tanh⁡′(X)\tanh'(X)
GTU's extra factor 1−tanh⁡2≤11-\tanh^2\le1, which vanishes when the value saturates

So GLU gives the gradient a linear path through open gates, like the identity path in an LSTM cell or a residual connection, while the gate still provides a nonlinearity. The GTU has to pass the gradient through two saturating functions.

Properties

Range
unbounded (the value is linear); the gate is in (0,1)(0,1)
Parameters
2 dindout2\,d_{in}d_{out} weights and 2dout2d_{out} biases: twice a linear layer of the same output width
Shape
nn.GLU(dim) maps size 2m2m to mm along dim
Not pointwise
each output is a function of two pre-activations; there is no single curve y=f(z)y=f(z) to plot
Smoothness
C∞C^\infty; the output is exactly 0 on the hyperplane a=0a=0, and near 0 wherever the gate is closed
Gradient
value path σ(g)≤1\sigma(g)\le1 (no tanh⁡′\tanh' factor); gate path aσ′(g)a\sigma'(g)
Family
replace σ with another function: Bilinear (identity), ReGLU, GEGLU, SwiGLU

Intuition and history

Gating is old: LSTMs (Hochreiter & Schmidhuber, 1997) use sigmoid gates to decide what to write, keep and read. Dauphin et al. (arXiv, December 2016; ICML 2017) stripped the idea down to one multiplicative gate per unit inside a stack of causal 1-D convolutions, h(X)=(X∗W+b)⊗σ(X∗V+c)h(X)=(X*W+b)\otimes\sigma(X*V+c). Their non-recurrent model reached state-of-the-art results on WikiText-103 and competitive results on Google Billion Word, and in their comparisons GLU beat the GTU as well as ungated tanh and ReLU convolutions.

The gate is input-dependent: the same unit can pass its value for one input and block it for another. In the figure the output is the value map with a sigmoid-shaped "window" laid over it; move the gate weights and the window slides and rotates. A ReLU can only zero a unit along its own pre-activation's sign; a GLU unit decides using a different direction in input space.

Shazeer (2020) swapped the sigmoid for other functions inside the Transformer feed-forward layer. The resulting family is now standard in large language models:

VariantfFormulaNotes
GLUσ\sigma(xW+b)⊙σ(xV+c)(xW+b)\odot\sigma(xV+c)Dauphin et al. 2016, gated conv LM
Bilinearidentity(xW)⊙(xV)(xW)\odot(xV)no nonlinearity except the product itself (Shazeer 2020)
ReGLUReLUmax⁡(0,xW)⊙xV\max(0,xW)\odot xVhard gate: closed means exactly zero gradient
GEGLUGELUGELU⁡(xW)⊙xV\operatorname{GELU}(xW)\odot xVT5 v1.1, Gemma; see GeGLU
SwiGLUSwish xσ(x)x\sigma(x)Swish⁡(xW)⊙xV\operatorname{Swish}(xW)\odot xVPaLM, LLaMA, Mistral; see SwiGLU
The GLU family, all of the form (xW)⊙f(⋅)(xW)\odot f(\cdot) with one side passed through ff
value a = w·x + bw = (0.8, 0.4), b = 0.1range [−2.20, 2.40]⊙gate σ(v·x + c)v = (−0.5, 1.0), c = 0.2range [0.06, 0.96]=output y = a ⊙ σ(g)GLUrange [−1.09, 1.83]■ positive ■ negative (scaled per map) · — a = 0 · - - g = 0 · x₁ →, x₂ ↑ ∈ [−2, 2]click or drag on any map to move the probe ○probe x = (1.0, −0.5): a = 0.700, g = −0.800, σ(g) = 0.310, y = 0.217∂y/∂a = σ(g) = 0.310 (value path) · ∂y/∂g = a·σ′(g) = 0.150 (gate path)GTU tanh(a)·σ(g) would give value-path slope tanh′(a)·σ(g) = 0.197
gate
fig.A single GLU unit on a 2-D input. Left: the value a = 0.8x₁ + 0.4x₂ + 0.1 (zero on the solid line). Middle: the gate, zero pre-activation on the dashed line. Right: their product. Slide the gate weights to move the window; switch the gate to ReLU (hard cut), identity (bilinear: a saddle) or GELU/Swish. Drag the probe to read off both gradient paths and compare with a tanh-valued GTU.
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 0.731 · f′(x) = 0.197012345−4−2024x →SigmoidSigmoid f′SwishGELUReLU
activation
compare
range (0, 1)monotonic yessmooth yeszero-centred no
Maximum slope 0.25 at x = 0; saturates for large |x|.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]0.020.500.98output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.500Var[f(x)] 0.043E[f′(x)] 0.207f′ = 0 for 0.0% of inputsE[f′]¹⁰ 1.4e−7
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.The gate nonlinearities of the GLU family, one at a time: sigmoid (GLU) saturates at 0 and 1; Swish, GELU and ReLU are unbounded above, so the gate can also amplify.

Worked example: one unit, forward and backward

Input x=(1, −0.5)x=(1,\,-0.5), value weights w=(0.8, 0.4)w=(0.8,\,0.4), b=0.1b=0.1, gate weights v=(−0.5, 1.0)v=(-0.5,\,1.0), c=0.2c=0.2 (the figure's defaults, probe at (1,−0.5)(1,-0.5)). Take the upstream gradient ∂L/∂y=1\partial\mathcal L/\partial y=1.

QuantityComputationValue
aa0.8⋅1+0.4⋅(−0.5)+0.10.8\cdot1+0.4\cdot(-0.5)+0.10.7000
gg−0.5⋅1+1.0⋅(−0.5)+0.2-0.5\cdot1+1.0\cdot(-0.5)+0.2−0.8000
σ(g)\sigma(g)1/(1+e0.8)1/(1+e^{0.8})0.3100
yya σ(g)a\,\sigma(g)0.2170
∂y/∂a=∂y/∂b\partial y/\partial a=\partial y/\partial bσ(g)\sigma(g)0.3100
∂y/∂g=∂y/∂c\partial y/\partial g=\partial y/\partial ca σ(g)(1−σ(g))=0.7⋅0.3100⋅0.6900a\,\sigma(g)(1-\sigma(g))=0.7\cdot0.3100\cdot0.69000.1497
∂y/∂w\partial y/\partial wσ(g) x\sigma(g)\,x(0.3100, −0.1550)
∂y/∂v\partial y/\partial vaσ′(g) xa\sigma'(g)\,x(0.1497, −0.0749)
∂y/∂x\partial y/\partial x0.3100 w+0.1497 v0.3100\,w+0.1497\,v(0.1732, 0.2737)
GLU forward and backward (4 decimals; finite differences agree)

A GTU with the same pre-activations gives tanh⁡(0.7) σ(−0.8)=0.1874\tanh(0.7)\,\sigma(-0.8)=0.1874 and value-path gain tanh⁡′(0.7) σ(g)=0.1968\tanh'(0.7)\,\sigma(g)=0.1968, already about a third below GLU's 0.3100. The gap grows with the value: at a=2a=2 the GTU's gain is 0.02190.0219 (a factor tanh⁡′(2)=0.0707\tanh'(2)=0.0707 smaller) and at a=3a=3 it is 0.00310.0031. GLU's value-path gain is σ(g)\sigma(g) regardless of aa.

Parameter count. A GLU layer from din=512d_{in}=512 to dout=512d_{out}=512 holds 2⋅512⋅512=524,2882\cdot512\cdot512=524{,}288 weights, twice a plain linear layer. Fair comparisons therefore shrink the gated layer's width; the GLU-variant FFNs in Transformers use 2/32/3 of the usual hidden size so three matrices cost what two did.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3glu = nn.GLU(dim=-1) # [A, B] -> A * sigmoid(B)
4print(glu(torch.tensor([1.0, -2.0, 0.5, 3.0]))) # [ 0.6225, -1.9051]
5
6class GLULayer(nn.Module):
7 def __init__(self, d_in, d_out, gate=torch.sigmoid):
8 super().__init__()
9 self.proj = nn.Linear(d_in, 2 * d_out) # value and gate in one matmul
10 self.gate = gate # sigmoid=GLU, F.relu=ReGLU, F.gelu=GEGLU, F.silu=SwiGLU
11 def forward(self, x):
12 a, g = self.proj(x).chunk(2, dim=-1)
13 return a * self.gate(g)
14
15layer = GLULayer(2, 1)
16with torch.no_grad():
17 layer.proj.weight.copy_(torch.tensor([[0.8, 0.4], [-0.5, 1.0]]))
18 layer.proj.bias.copy_(torch.tensor([0.1, 0.2]))
19x = torch.tensor([[1.0, -0.5]], requires_grad=True)
20y = layer(x); y.sum().backward()
21print(y) # [[0.2170]]
22print(layer.proj.weight.grad) # [[0.3100, -0.1550], [0.1497, -0.0749]]
23print(x.grad) # [[0.1732, 0.2737]]
nn.GLU splits and gates; a from-scratch module reproduces the worked example

Use when

  • +Input-dependent gating with a linear gradient path through open gates.
  • +Multiplicative feature interactions in a single layer.
  • +Consistently better than ReLU/GELU feed-forward layers at equal parameters in Transformer language models (Shazeer 2020).
  • +Trivial to implement: one wider matmul, a split, a pointwise product.

Avoid when

  • −Twice the weights of a linear layer of the same width; width must be cut to compare fairly.
  • −Three matrices in an FFN instead of two: more kernel launches and activation memory per token.
  • −Unbounded output and an output that can be large where both aa and the gate are large.
  • −No theory says why it works better; the evidence is empirical.

Check yourself

A GLU unit has value a=3a=3 and an open gate σ(g)=0.9\sigma(g)=0.9. What is ∂y/∂a\partial y/\partial a, and what would a GTU, tanh⁡(a)σ(g)\tanh(a)\sigma(g), give?

Check yourself

nn.GLU(dim=-1) receives a tensor of shape (8, 128, 1024). What shape comes out?

Remember

  • ✓GLU⁡(x)=(xW+b)⊙σ(xV+c)\operatorname{GLU}(x)=(xW+b)\odot\sigma(xV+c): a linear value times a sigmoid gate (Dauphin et al., 2016).
  • ✓Gradient: value path σ(g)\sigma(g), gate path aσ′(g)a\sigma'(g); no tanh⁡′\tanh' factor, unlike the GTU.
  • ✓Twice the weights of a linear layer; nn.GLU splits a 2m2m-wide input into mm outputs.
  • ✓Gates are input-dependent and multiplicative: one layer can express products of features.
  • ✓Family: Bilinear, ReGLU, GEGLU, SwiGLU; the FFN versions use 23\tfrac23 of the hidden width to match parameters.
  • ✓Worked example: a=0.7a=0.7, σ(g)=0.31\sigma(g)=0.31, y=0.217y=0.217, ∂y/∂g=0.150\partial y/\partial g=0.150.
← back to Deep Learning roadmap