# GLU (Gated Linear Unit)

> Two linear projections, one passed through a sigmoid and multiplied into the other: an input-dependent gate with a linear path for the gradient.

Source: https://www.mlfoundry.in/learn/glu-gated-linear-unit  
Section: Deep Learning › Activation Functions  
Level: advanced · 9 min read · updated 2026-10-11

**Gated Linear Unit (GLU):** is the layer $\operatorname{GLU}(x)=(xW+b)\odot\sigma(xV+c)$: one linear projection (the **value**) multiplied elementwise by a sigmoid of a second linear projection (the **gate**). It was introduced by Dauphin, Fan, Auli and Grangier (2016) for gated convolutional language models.

An ordinary activation bends one pre-activation: $y=\varphi(z)$. A GLU computes **two** pre-activations from the same input and lets one decide how much of the other gets through. Each output unit is a product of two linear functions of $x$ (one squashed into $(0,1)$), so the layer can express multiplicative interactions that a single linear map followed by a pointwise nonlinearity cannot.

The value path stays linear: there is no squashing function on $xW+b$. That is the point of the design, and it is what separates GLU from the older tanh-based gate.

## Formula

$$
\operatorname{GLU}(x)=\underbrace{(xW+b)}_{a\ \text{(value)}}\;\odot\;\sigma\big(\underbrace{xV+c}_{g\ \text{(gate)}}\big),\qquad W,V\in\mathbb{R}^{d_{in}\times d_{out}}
$$

- $a=xW+b$: value: a plain linear projection, unbounded
- $g=xV+c$: gate pre-activation, a second projection of the same input
- $\sigma$: logistic sigmoid $1/(1+e^{-g})$, so each gate is a soft switch in $(0,1)$
- $\odot$: elementwise product: unit $k$ outputs $a_k\,\sigma(g_k)$

In code the two projections are usually fused into one matrix of width $2d_{out}$ whose output is split in half. `torch.nn.GLU(dim=-1)` does only the split-and-gate step: for an input $[A,\,B]$ along `dim` it returns $A\odot\sigma(B)$, halving that dimension.

## Gradient

By the product rule, the gradient of one unit splits into a **value path** and a **gate path**:

$$
\frac{\partial y}{\partial a}=\sigma(g),\qquad \frac{\partial y}{\partial g}=a\,\sigma(g)\big(1-\sigma(g)\big),\qquad \frac{\partial y}{\partial x}=\sigma(g)\,W_{:,k}+a\,\sigma'(g)\,V_{:,k}
$$

- $\sigma(g)$: value-path gain: the gate itself, no derivative of a squashing function
- $a\,\sigma'(g)$: gate-path gain, at most $|a|/4$
- $W_{:,k},V_{:,k}$: column $k$ of each weight matrix

Dauphin et al. write it for a whole layer and contrast it with the gated tanh unit (GTU) $\tanh(X)\odot\sigma(X)$ used in WaveNet and PixelCNN (van den Oord et al., 2016):

$$
\nabla[X\otimes\sigma(X)]=\nabla X\otimes\sigma(X)+X\otimes\sigma'(X)\nabla X\qquad\text{vs}\qquad \nabla[\tanh(X)\otimes\sigma(X)]=\tanh'(X)\nabla X\otimes\sigma(X)+\sigma'(X)\nabla X\otimes\tanh(X)
$$

- $\nabla X\otimes\sigma(X)$: GLU's first term: the gradient flows through any open gate unscaled
- $\tanh'(X)$: GTU's extra factor $1-\tanh^2\le1$, which vanishes when the value saturates

So GLU gives the gradient a linear path through open gates, like the identity path in an LSTM cell or a residual connection, while the gate still provides a nonlinearity. The GTU has to pass the gradient through two saturating functions.

**Properties**

- **Range:** unbounded (the value is linear); the gate is in $(0,1)$
- **Parameters:** $2\,d_{in}d_{out}$ weights and $2d_{out}$ biases: twice a linear layer of the same output width
- **Shape:** `nn.GLU(dim)` maps size $2m$ to $m$ along `dim`
- **Not pointwise:** each output is a function of two pre-activations; there is no single curve $y=f(z)$ to plot
- **Smoothness:** $C^\infty$; the output is exactly 0 on the hyperplane $a=0$, and near 0 wherever the gate is closed
- **Gradient:** value path $\sigma(g)\le1$ (no $\tanh'$ factor); gate path $a\sigma'(g)$
- **Family:** replace σ with another function: Bilinear (identity), ReGLU, GEGLU, SwiGLU

## Intuition and history

Gating is old: LSTMs (Hochreiter & Schmidhuber, 1997) use sigmoid gates to decide what to write, keep and read. Dauphin et al. (arXiv, December 2016; ICML 2017) stripped the idea down to one multiplicative gate per unit inside a stack of causal 1-D convolutions, $h(X)=(X*W+b)\otimes\sigma(X*V+c)$. Their non-recurrent model reached state-of-the-art results on WikiText-103 and competitive results on Google Billion Word, and in their comparisons GLU beat the GTU as well as ungated tanh and ReLU convolutions.

The gate is **input-dependent**: the same unit can pass its value for one input and block it for another. In the figure the output is the value map with a sigmoid-shaped "window" laid over it; move the gate weights and the window slides and rotates. A ReLU can only zero a unit along its own pre-activation's sign; a GLU unit decides using a different direction in input space.

Shazeer (2020) swapped the sigmoid for other functions inside the Transformer feed-forward layer. The resulting family is now standard in large language models:

The GLU family, all of the form $(xW)\odot f(\cdot)$ with one side passed through $f$

| Variant | f | Formula | Notes |
| --- | --- | --- | --- |
| GLU | $\sigma$ | $(xW+b)\odot\sigma(xV+c)$ | Dauphin et al. 2016, gated conv LM |
| Bilinear | identity | $(xW)\odot(xV)$ | no nonlinearity except the product itself (Shazeer 2020) |
| ReGLU | ReLU | $\max(0,xW)\odot xV$ | hard gate: closed means exactly zero gradient |
| GEGLU | GELU | $\operatorname{GELU}(xW)\odot xV$ | T5 v1.1, Gemma; see **GeGLU** |
| SwiGLU | Swish $x\sigma(x)$ | $\operatorname{Swish}(xW)\odot xV$ | PaLM, LLaMA, Mistral; see **SwiGLU** |

> Figure: A single GLU unit on a 2-D input. Left: the value a = 0.8x₁ + 0.4x₂ + 0.1 (zero on the solid line). Middle: the gate, zero pre-activation on the dashed line. Right: their product. Slide the gate weights to move the window; switch the gate to ReLU (hard cut), identity (bilinear: a saddle) or GELU/Swish. Drag the probe to read off both gradient paths and compare with a tanh-valued GTU.

> Figure: The gate nonlinearities of the GLU family, one at a time: sigmoid (GLU) saturates at 0 and 1; Swish, GELU and ReLU are unbounded above, so the gate can also amplify.

## Worked example: one unit, forward and backward

Input $x=(1,\,-0.5)$, value weights $w=(0.8,\,0.4)$, $b=0.1$, gate weights $v=(-0.5,\,1.0)$, $c=0.2$ (the figure's defaults, probe at $(1,-0.5)$). Take the upstream gradient $\partial\mathcal L/\partial y=1$.

GLU forward and backward (4 decimals; finite differences agree)

| Quantity | Computation | Value |
| --- | --- | --- |
| $a$ | $0.8\cdot1+0.4\cdot(-0.5)+0.1$ | 0.7000 |
| $g$ | $-0.5\cdot1+1.0\cdot(-0.5)+0.2$ | −0.8000 |
| $\sigma(g)$ | $1/(1+e^{0.8})$ | 0.3100 |
| $y$ | $a\,\sigma(g)$ | 0.2170 |
| $\partial y/\partial a=\partial y/\partial b$ | $\sigma(g)$ | 0.3100 |
| $\partial y/\partial g=\partial y/\partial c$ | $a\,\sigma(g)(1-\sigma(g))=0.7\cdot0.3100\cdot0.6900$ | 0.1497 |
| $\partial y/\partial w$ | $\sigma(g)\,x$ | (0.3100, −0.1550) |
| $\partial y/\partial v$ | $a\sigma'(g)\,x$ | (0.1497, −0.0749) |
| $\partial y/\partial x$ | $0.3100\,w+0.1497\,v$ | (0.1732, 0.2737) |

A GTU with the same pre-activations gives $\tanh(0.7)\,\sigma(-0.8)=0.1874$ and value-path gain $\tanh'(0.7)\,\sigma(g)=0.1968$, already about a third below GLU's 0.3100. The gap grows with the value: at $a=2$ the GTU's gain is $0.0219$ (a factor $\tanh'(2)=0.0707$ smaller) and at $a=3$ it is $0.0031$. GLU's value-path gain is $\sigma(g)$ regardless of $a$.

**Parameter count.** A GLU layer from $d_{in}=512$ to $d_{out}=512$ holds $2\cdot512\cdot512=524{,}288$ weights, twice a plain linear layer. Fair comparisons therefore shrink the gated layer's width; the GLU-variant FFNs in Transformers use $2/3$ of the usual hidden size so three matrices cost what two did.

```python
import torch, torch.nn as nn, torch.nn.functional as F

glu = nn.GLU(dim=-1)                        # [A, B] -> A * sigmoid(B)
print(glu(torch.tensor([1.0, -2.0, 0.5, 3.0])))   # [ 0.6225, -1.9051]

class GLULayer(nn.Module):
    def __init__(self, d_in, d_out, gate=torch.sigmoid):
        super().__init__()
        self.proj = nn.Linear(d_in, 2 * d_out)  # value and gate in one matmul
        self.gate = gate                        # sigmoid=GLU, F.relu=ReGLU, F.gelu=GEGLU, F.silu=SwiGLU
    def forward(self, x):
        a, g = self.proj(x).chunk(2, dim=-1)
        return a * self.gate(g)

layer = GLULayer(2, 1)
with torch.no_grad():
    layer.proj.weight.copy_(torch.tensor([[0.8, 0.4], [-0.5, 1.0]]))
    layer.proj.bias.copy_(torch.tensor([0.1, 0.2]))
x = torch.tensor([[1.0, -0.5]], requires_grad=True)
y = layer(x); y.sum().backward()
print(y)                       # [[0.2170]]
print(layer.proj.weight.grad)  # [[0.3100, -0.1550], [0.1497, -0.0749]]
print(x.grad)                  # [[0.1732, 0.2737]]
```

nn.GLU splits and gates; a from-scratch module reproduces the worked example

> **Which half is the gate?:** `nn.GLU` gates the **first** half with the sigmoid of the **second**. Other code bases (e.g. Hugging Face's `gate_proj`/`up_proj` LLaMA layers, or diffusers' `GEGLU`) use separate or differently ordered halves. When porting weights between implementations, check the order or the model silently computes $\sigma(a)\odot g$.

> **Width doubles the cost:** Dropping a GLU in place of a ReLU layer of the same output width doubles that layer's weights and FLOPs. Gains reported for GLU variants are at matched parameter count; compare at equal cost or you are mostly measuring the extra capacity.

> **Gating is multiplication, and multiplication is expressive:** With the identity gate, each output $(w\cdot x)(v\cdot x)$ is a quadratic form in $x$: the figure's bilinear surface is a saddle. Products of features are hard to build from sums and pointwise nonlinearities alone; a gate gets them in one layer. Attention, LSTM gates and FiLM conditioning all rely on the same multiplicative trick.

**Strengths**

- Input-dependent gating with a linear gradient path through open gates.
- Multiplicative feature interactions in a single layer.
- Consistently better than ReLU/GELU feed-forward layers at equal parameters in Transformer language models (Shazeer 2020).
- Trivial to implement: one wider matmul, a split, a pointwise product.

**Limitations**

- Twice the weights of a linear layer of the same width; width must be cut to compare fairly.
- Three matrices in an FFN instead of two: more kernel launches and activation memory per token.
- Unbounded output and an output that can be large where both $a$ and the gate are large.
- No theory says why it works better; the evidence is empirical.

**Question:** A GLU unit has value $a=3$ and an open gate $\sigma(g)=0.9$. What is $\partial y/\partial a$, and what would a GTU, $\tanh(a)\sigma(g)$, give?

- A. 0.9 for GLU, about 0.009 for GTU
- B. 0.9 for both
- C. 2.7 for GLU, 0.9 for GTU
- D. 0.09 for GLU, 0.9 for GTU

**Answer:** A. 0.9 for GLU, about 0.009 for GTU — For GLU $\partial y/\partial a=\sigma(g)=0.9$ whatever $a$ is. The GTU multiplies by $\tanh'(3)=1-\tanh^2(3)=0.0099$, giving about $0.0089$: the saturated tanh almost blocks the value path. This is the "linear path" argument of Dauphin et al.

**Question:** `nn.GLU(dim=-1)` receives a tensor of shape (8, 128, 1024). What shape comes out?

- A. (8, 128, 1024)
- B. (8, 128, 512)
- C. (8, 128, 2048)
- D. (8, 64, 1024)

**Answer:** B. (8, 128, 512) — The last dimension is split into two halves of 512, $A$ and $B$, and the output is $A\odot\sigma(B)$, so the last dimension halves. To get a 1024-wide output you project to 2048 first.

**Key points**

- $\operatorname{GLU}(x)=(xW+b)\odot\sigma(xV+c)$: a linear value times a sigmoid gate (Dauphin et al., 2016).
- Gradient: value path $\sigma(g)$, gate path $a\sigma'(g)$; no $\tanh'$ factor, unlike the GTU.
- Twice the weights of a linear layer; `nn.GLU` splits a $2m$-wide input into $m$ outputs.
- Gates are input-dependent and multiplicative: one layer can express products of features.
- Family: Bilinear, ReGLU, GEGLU, SwiGLU; the FFN versions use $\tfrac23$ of the hidden width to match parameters.
- Worked example: $a=0.7$, $\sigma(g)=0.31$, $y=0.217$, $\partial y/\partial g=0.150$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
