# GeGLU

> A GELU-gated GLU feed-forward layer, (GELU(xW) ⊙ xV)W₂: best in Shazeer's 2020 comparison and used in T5 v1.1, Gemma and Stable Diffusion's transformer blocks.

Source: https://www.mlfoundry.in/learn/geglu  
Section: Deep Learning › Activation Functions  
Level: advanced · 8 min read · updated 2026-10-11

**GeGLU:** (also written GEGLU) is the gated feed-forward block $\operatorname{FFN}_{\text{GEGLU}}(x)=\big(\operatorname{GELU}(xW)\odot xV\big)W_2$: a GLU whose gate is the Gaussian Error Linear Unit $\operatorname{GELU}(z)=z\,\Phi(z)$. Like SwiGLU it comes from Shazeer, *GLU Variants Improve Transformer* (2020).

GeGLU and SwiGLU are siblings: same three-matrix block, same $\tfrac23$ hidden-width rule, different gate curve. In Shazeer's experiments the two finished essentially tied at the top. Model families tend to pick one and stick with it: Google's T5 v1.1 and Gemma use GeGLU, Meta's LLaMA line uses SwiGLU.

This article focuses on what the gate function changes inside the product: the shape of the output, and how gradient is split between the gate and value paths.

## Formula

$$
\operatorname{FFN}_{\text{GEGLU}}(x)=\Big(\operatorname{GELU}(xW)\odot xV\Big)W_2,\qquad \operatorname{GELU}(z)=z\,\Phi(z)=\tfrac{z}{2}\Big(1+\operatorname{erf}\big(z/\sqrt2\big)\Big)
$$

- $W,V\in\mathbb{R}^{d\times h}$: gate and value projections; T5 v1.1 calls them `wi_0` and `wi_1`
- $W_2\in\mathbb{R}^{h\times d}$: output projection (`wo`)
- $\Phi$: standard normal CDF
- $h$: $\tfrac23$ of a standard FFN's width at equal parameters, e.g. 2048 instead of 3072 for $d=768$

$$
\operatorname{GELU}_{\tanh}(z)=\tfrac{z}{2}\Big(1+\tanh\!\Big(\sqrt{2/\pi}\,\big(z+0.044715\,z^3\big)\Big)\Big)
$$

- $\operatorname{GELU}_{\tanh}$: the tanh approximation (Hendrycks & Gimpel 2016), `F.gelu(x, approximate="tanh")`; within $4.7\times10^{-4}$ of the exact GELU everywhere (worst near $|z|=2.70$). Gemma's "approximated GeGLU" uses it

## Gradient

With gate pre-activation $g=xW$ and value $u=xV$, each hidden unit computes $y_k=\operatorname{GELU}(g_k)\,u_k$:

$$
\frac{\partial y_k}{\partial u_k}=\operatorname{GELU}(g_k),\qquad \frac{\partial y_k}{\partial g_k}=u_k\,\operatorname{GELU}'(g_k),\qquad \operatorname{GELU}'(z)=\Phi(z)+z\,\varphi(z)
$$

- $\varphi$: standard normal density $e^{-z^2/2}/\sqrt{2\pi}$
- $\operatorname{GELU}'$: 0.5 at 0, peak 1.1289 at $z=\sqrt2$, negative for $z<-0.7518$ (left of GELU's minimum $-0.1700$)

**Properties**

- **Gate:** $z\Phi(z)$: smooth, non-monotone, minimum $-0.1700$ at $z=-0.7518$, unbounded above
- **vs Swish:** $\max_z|\operatorname{GELU}(z)-z\sigma(z)|=0.193$ at $|z|\approx1.97$; vs $z\sigma(1.702z)$ the gap is only $0.020$
- **Parameters:** $3dh$, same as SwiGLU; equal to a $4d$ FFN at $h=\tfrac83d$
- **Gradient:** value path $\operatorname{GELU}(g)$, gate path $u\,\operatorname{GELU}'(g)$; both nonzero for negative $g$
- **Shazeer 2020:** log-perplexity 1.942 / 1.633 (65k / 524k steps), the best of the eight FFNs compared; SwiGLU 1.944 / 1.636, ReLU 1.997 / 1.677
- **Used in:** T5 v1.1 and mT5, Gemma (tanh-approximated), the transformer blocks of Stable Diffusion's UNet (`GEGLU` in diffusers and the original latent-diffusion code)

## GELU vs Swish vs ReLU inside the gate

GELU and Swish look alike, but $\beta=1$ Swish is a noticeably softer curve: the largest gap is $0.193$ at $|z|\approx1.97$ (at $z=-2$: GELU $-0.046$, Swish $-0.238$). Their derivatives differ by up to $0.160$ near $|z|\approx0.87$. GELU is much closer to $z\sigma(1.702z)$, i.e. Swish with $\beta=1.702$ (max gap $0.020$). Shazeer's results put the two gates within noise of each other, so the choice matters less than the gating itself.

ReLU (ReGLU) is the hard version: for $g<0$ the gate is exactly zero, so the output **and both gradients** vanish for that unit and that token. Sigmoid (the original GLU) never closes fully but also never exceeds 1, so it can only attenuate; GELU and Swish can amplify a value by more than its own size when the gate is large.

> Figure: Top: the GLU product y = gate(g)·v as the gate pre-activation g sweeps from −4 to 4, for a fixed value v. Bottom: the gate-path gradient v·gate′(g). The table gives both gradients at the slider's g. Note where GELU and Swish differ (around |g| ≈ 2, and in sign of the slope near g = −1), and how ReLU's gradient is a step. Flip v negative to mirror everything; toggle the tanh approximation to see it is visually identical.

> Figure: GELU, the GeGLU gate, beside Swish (SwiGLU) and ReLU (ReGLU).

## History and use

GELU was proposed by Hendrycks and Gimpel (2016) as a smooth, probabilistic ReLU and became the FFN activation of BERT and GPT-2. Shazeer (2020) used it as a gate and got the best numbers of his comparison. Google's T5 v1.1 checkpoints replaced ReLU with GeGLU in the FFN (with $d_{ff}=2048$ for the base size), and mT5 follows that recipe. Gemma (2024) uses the tanh-approximated GeGLU.

Latent diffusion / Stable Diffusion put a GeGLU FFN in every spatial transformer block of the UNet. That implementation keeps the inner width at $4\times$ the channel count, so it is **not** parameter-matched: for 320 channels it holds $320\cdot2560+1280\cdot320=1{,}228{,}800$ weights against $819{,}200$ for a plain $4\times$ FFN, 1.5× as many.

## Worked example: three hidden units, three gates

The same inputs as in the SwiGLU article: gate pre-activations $g=(2.0,\,-1.0,\,0.5)$, values $u=(1.5,\,2.0,\,-1.0)$, upstream gradient 1.

Output y and gate-path gradient ∂y/∂g = u·f′(g); the value-path gradient ∂y/∂u is the gate value f(g) (4 decimals)

| Gate f | f(g) | y = f(g)·u | ∂y/∂g |
| --- | --- | --- | --- |
| GELU (GeGLU) | (1.9545, −0.1587, 0.3457) | (2.9317, −0.3173, −0.3457) | (1.6278, −0.1666, −0.8675) |
| Swish (SwiGLU) | (1.7616, −0.2689, 0.3112) | (2.6424, −0.5379, −0.3112) | (1.6362, 0.1447, −0.7400) |
| ReLU (ReGLU) | (2, 0, 0.5) | (3, 0, −0.5) | (1.5, 0, −1.0) |
| sigmoid (GLU) | (0.8808, 0.2689, 0.6225) | (1.3212, 0.5379, −0.6225) | (0.1575, 0.3932, −0.2350) |

Unit 2 ($g=-1$) separates the gates. GELU's slope at $-1$ is $\operatorname{GELU}'(-1)=-0.0833$, so its gate gradient is $-0.167$; Swish's slope is $+0.0723$, giving $+0.145$. The two push $g_2$ in **opposite** directions, because $-1$ lies left of GELU's minimum ($-0.75$) and right of Swish's ($-1.28$). ReLU sends nothing at all. With the tanh approximation GeGLU's outputs become $(2.9319,\,-0.3176,\,-0.3457)$.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class GeGLUFFN(nn.Module):                     # T5 v1.1 naming: wi_0 gate, wi_1 value, wo out
    def __init__(self, d, h, approximate="none"):
        super().__init__()
        self.wi_0 = nn.Linear(d, h, bias=False)
        self.wi_1 = nn.Linear(d, h, bias=False)
        self.wo = nn.Linear(h, d, bias=False)
        self.approximate = approximate           # "tanh" for Gemma-style GeGLU
    def forward(self, x):
        return self.wo(F.gelu(self.wi_0(x), approximate=self.approximate) * self.wi_1(x))

class GEGLU(nn.Module):                        # diffusers-style: one fused projection
    def __init__(self, d_in, d_out):
        super().__init__()
        self.proj = nn.Linear(d_in, 2 * d_out)
    def forward(self, x):
        value, gate = self.proj(x).chunk(2, dim=-1)
        return value * F.gelu(gate)

ffn = GeGLUFFN(768, 2048)
print(sum(p.numel() for p in ffn.parameters()))   # 4718592 = 2 * 768 * 3072

g = torch.tensor([2.0, -1.0, 0.5], requires_grad=True)
u = torch.tensor([1.5, 2.0, -1.0], requires_grad=True)
y = F.gelu(g) * u
y.sum().backward()
print(y)        # [ 2.9317, -0.3173, -0.3457]
print(g.grad)   # [ 1.6278, -0.1666, -0.8675]
print(F.gelu(g, approximate="tanh") * u)          # [ 2.9319, -0.3176, -0.3457]
```

A T5 v1.1-style GeGLU FFN, the diffusers-style fused projection, and the worked example

> **Exact vs tanh GELU when loading checkpoints:** The two forms differ by at most $4.7\times10^{-4}$ per element, which is harmless for training but can shift logits measurably in a deep stack. Use the variant the checkpoint was trained with: `gelu_pytorch_tanh` for Gemma in Hugging Face configs, exact `gelu` or `gelu_new` (tanh) as each model's config specifies.

> **GeGLU is not GELU:** Replacing `nn.GELU()` in an MLP with "GeGLU" means adding a second projection and multiplying, not changing one function. Count the extra matrix: at the same hidden width the block has 1.5× the weights, as in Stable Diffusion's UNet.

> **The gate shape matters less than the gate:** In Shazeer's table every gated variant beats every ungated one, while the spread among GEGLU, SwiGLU and ReGLU (1.633–1.645 at 524k steps) is smaller than the gap to the best ungated FFN (1.677). The multiplicative interaction does most of the work; the smooth gate adds a little more.

**Strengths**

- Best log-perplexity in Shazeer's parameter-matched comparison, essentially tied with SwiGLU.
- Smooth gate with nonzero gradient for negative pre-activations (unlike ReGLU).
- Drop-in FFN replacement with the 2/3 width rule; mature fused kernels.
- A tanh approximation is available that is accurate to 5e-4.

**Limitations**

- Exact GELU needs erf; the tanh form is cheaper but must match the checkpoint.
- Three matrices and two h-wide activations to store, like SwiGLU.
- Advantages over SwiGLU are within noise; the choice is mostly convention.
- No theoretical account of why gating with GELU helps.

**Question:** A GeGLU unit has gate pre-activation $g=-3$ and value $u=2$. Roughly how much gradient reaches $u$, and how does that compare with ReGLU?

- A. About −0.004 for GeGLU; exactly 0 for ReGLU
- B. Exactly 0 for both
- C. About 2 for both
- D. About −0.5 for GeGLU; 2 for ReGLU

**Answer:** A. About −0.004 for GeGLU; exactly 0 for ReGLU — $\partial y/\partial u=\operatorname{GELU}(g)=-3\,\Phi(-3)=-3\cdot0.00135\approx-0.004$: tiny but nonzero. ReLU's gate is exactly 0 for $g<0$, so ReGLU passes no gradient to $u$ or $g$.

**Question:** Which statement about GELU and β = 1 Swish as gates is correct?

- A. They are identical functions
- B. They differ by up to about 0.19, and their slopes can have opposite signs near g = −1
- C. Swish is bounded above, GELU is not
- D. GELU has no negative outputs

**Answer:** B. They differ by up to about 0.19, and their slopes can have opposite signs near g = −1 — $\max|z\Phi(z)-z\sigma(z)|=0.193$ at $|z|\approx1.97$. At $g=-1$, $\operatorname{GELU}'=-0.083$ while $\operatorname{Swish}'=+0.072$. Both are unbounded above and both dip below zero (minima $-0.170$ and $-0.278$).

**Key points**

- $\operatorname{FFN}_{\text{GEGLU}}(x)=(\operatorname{GELU}(xW)\odot xV)W_2$ (Shazeer 2020).
- Gradients: $\partial y/\partial u=\operatorname{GELU}(g)$, $\partial y/\partial g=u\,(\Phi(g)+g\varphi(g))$.
- Best in Shazeer's table (1.633 at 524k steps), within noise of SwiGLU (1.636).
- GELU vs Swish: max gap 0.193 at $|z|\approx1.97$; ReLU gating zeroes output and gradient for $g<0$.
- Used in T5 v1.1, mT5, Gemma (tanh form) and Stable Diffusion's UNet transformer blocks (at 4× width).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
