GeGLU

A GELU-gated GLU feed-forward layer, (GELU(xW) ⊙ xV)W₂: best in Shazeer's 2020 comparison and used in T5 v1.1, Gemma and Stable Diffusion's transformer blocks.

8 min read629 wordsadvanced0 viewsupdated 2026-10-11

Definition

GeGLU (also written GEGLU) is the gated feed-forward block FFN⁡GEGLU(x)=(GELU⁡(xW)⊙xV)W2\operatorname{FFN}_{\text{GEGLU}}(x)=\big(\operatorname{GELU}(xW)\odot xV\big)W_2: a GLU whose gate is the Gaussian Error Linear Unit GELU⁡(z)=z Φ(z)\operatorname{GELU}(z)=z\,\Phi(z). Like SwiGLU it comes from Shazeer, GLU Variants Improve Transformer (2020).

GeGLU and SwiGLU are siblings: same three-matrix block, same 23\tfrac23 hidden-width rule, different gate curve. In Shazeer's experiments the two finished essentially tied at the top. Model families tend to pick one and stick with it: Google's T5 v1.1 and Gemma use GeGLU, Meta's LLaMA line uses SwiGLU.

This article focuses on what the gate function changes inside the product: the shape of the output, and how gradient is split between the gate and value paths.

Formula

FFN⁡GEGLU(x)=(GELU⁡(xW)⊙xV)W2,GELU⁡(z)=z Φ(z)=z2(1+erf⁡(z/2))\operatorname{FFN}_{\text{GEGLU}}(x)=\Big(\operatorname{GELU}(xW)\odot xV\Big)W_2,\qquad \operatorname{GELU}(z)=z\,\Phi(z)=\tfrac{z}{2}\Big(1+\operatorname{erf}\big(z/\sqrt2\big)\Big)
W,V∈Rd×hW,V\in\mathbb{R}^{d\times h}
gate and value projections; T5 v1.1 calls them wi_0 and wi_1
W2∈Rh×dW_2\in\mathbb{R}^{h\times d}
output projection (wo)
Φ\Phi
standard normal CDF
hh
23\tfrac23 of a standard FFN's width at equal parameters, e.g. 2048 instead of 3072 for d=768d=768
GELU⁡tanh⁡(z)=z2(1+tanh⁡ ⁣(2/π (z+0.044715 z3)))\operatorname{GELU}_{\tanh}(z)=\tfrac{z}{2}\Big(1+\tanh\!\Big(\sqrt{2/\pi}\,\big(z+0.044715\,z^3\big)\Big)\Big)
GELU⁡tanh⁡\operatorname{GELU}_{\tanh}
the tanh approximation (Hendrycks & Gimpel 2016), F.gelu(x, approximate="tanh"); within 4.7×10−44.7\times10^{-4} of the exact GELU everywhere (worst near ∣z∣=2.70|z|=2.70). Gemma's "approximated GeGLU" uses it

Gradient

With gate pre-activation g=xWg=xW and value u=xVu=xV, each hidden unit computes yk=GELU⁡(gk) uky_k=\operatorname{GELU}(g_k)\,u_k:

∂yk∂uk=GELU⁡(gk),∂yk∂gk=uk GELU⁡′(gk),GELU⁡′(z)=Φ(z)+z φ(z)\frac{\partial y_k}{\partial u_k}=\operatorname{GELU}(g_k),\qquad \frac{\partial y_k}{\partial g_k}=u_k\,\operatorname{GELU}'(g_k),\qquad \operatorname{GELU}'(z)=\Phi(z)+z\,\varphi(z)
φ\varphi
standard normal density e−z2/2/2πe^{-z^2/2}/\sqrt{2\pi}
GELU⁡′\operatorname{GELU}'
0.5 at 0, peak 1.1289 at z=2z=\sqrt2, negative for z<−0.7518z<-0.7518 (left of GELU's minimum −0.1700-0.1700)

Properties

Gate
zΦ(z)z\Phi(z): smooth, non-monotone, minimum −0.1700-0.1700 at z=−0.7518z=-0.7518, unbounded above
vs Swish
max⁡z∣GELU⁡(z)−zσ(z)∣=0.193\max_z|\operatorname{GELU}(z)-z\sigma(z)|=0.193 at ∣z∣≈1.97|z|\approx1.97; vs zσ(1.702z)z\sigma(1.702z) the gap is only 0.0200.020
Parameters
3dh3dh, same as SwiGLU; equal to a 4d4d FFN at h=83dh=\tfrac83d
Gradient
value path GELU⁡(g)\operatorname{GELU}(g), gate path u GELU⁡′(g)u\,\operatorname{GELU}'(g); both nonzero for negative gg
Shazeer 2020
log-perplexity 1.942 / 1.633 (65k / 524k steps), the best of the eight FFNs compared; SwiGLU 1.944 / 1.636, ReLU 1.997 / 1.677
Used in
T5 v1.1 and mT5, Gemma (tanh-approximated), the transformer blocks of Stable Diffusion's UNet (GEGLU in diffusers and the original latent-diffusion code)

GELU vs Swish vs ReLU inside the gate

GELU and Swish look alike, but β=1\beta=1 Swish is a noticeably softer curve: the largest gap is 0.1930.193 at ∣z∣≈1.97|z|\approx1.97 (at z=−2z=-2: GELU −0.046-0.046, Swish −0.238-0.238). Their derivatives differ by up to 0.1600.160 near ∣z∣≈0.87|z|\approx0.87. GELU is much closer to zσ(1.702z)z\sigma(1.702z), i.e. Swish with β=1.702\beta=1.702 (max gap 0.0200.020). Shazeer's results put the two gates within noise of each other, so the choice matters less than the gating itself.

ReLU (ReGLU) is the hard version: for g<0g<0 the gate is exactly zero, so the output and both gradients vanish for that unit and that token. Sigmoid (the original GLU) never closes fully but also never exceeds 1, so it can only attenuate; GELU and Swish can amplify a value by more than its own size when the gate is large.

y = gate(g) · v, v = 1.000−4−2024gate-path gradient ∂y/∂g = v · gate′(g)−4−2024−0.64.3−0.21.2gate pre-activation g = (xW)ᵢ →at g = 0.500, v = 1.000gatey∂y/∂v∂y/∂gGELU0.3460.3460.867Swish0.3110.3110.740ReLU0.5000.5001.000σ0.6220.6220.235∂y/∂v = gate(g): value-path gain∂y/∂g = v·gate′(g): gate-path gain|GELU − Swish| here: 0.035largest gap: 0.193 at |g| ≈ 1.97ReLU, g < 0: both gradients exactly 0GELU min: −0.170 at g ≈ −0.75Swish min: −0.278 at g ≈ −1.28
GELU form■ GeGLU■ SwiGLU■ ReGLU■ GLU
fig.Top: the GLU product y = gate(g)·v as the gate pre-activation g sweeps from −4 to 4, for a fixed value v. Bottom: the gate-path gradient v·gate′(g). The table gives both gradients at the slider's g. Note where GELU and Swish differ (around |g| ≈ 2, and in sign of the slope near g = −1), and how ReLU's gradient is a step. Flip v negative to mirror everything; toggle the tanh approximation to see it is visually identical.
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 0.841 · f′(x) = 1.083−1012345−4−2024x →GELU (exact)GELU f′SwishReLU
activation
compare
form
range [≈ −0.170, ∞)monotonic nosmooth yeszero-centred no
Exact form by default. approx = 1 switches to ½x(1 + tanh(√(2/π)(x + 0.044715x³))), max error ≈ 5e-4. Minimum ≈ −0.170 at x ≈ −0.752.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no sampling↑ 40%E[f]−0.171.914.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.282Var[f(x)] 0.346E[f′(x)] 0.500f′ = 0 for 0.0% of inputsE[f′]¹⁰ 9.8e−4
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.GELU, the GeGLU gate, beside Swish (SwiGLU) and ReLU (ReGLU).

History and use

GELU was proposed by Hendrycks and Gimpel (2016) as a smooth, probabilistic ReLU and became the FFN activation of BERT and GPT-2. Shazeer (2020) used it as a gate and got the best numbers of his comparison. Google's T5 v1.1 checkpoints replaced ReLU with GeGLU in the FFN (with dff=2048d_{ff}=2048 for the base size), and mT5 follows that recipe. Gemma (2024) uses the tanh-approximated GeGLU.

Latent diffusion / Stable Diffusion put a GeGLU FFN in every spatial transformer block of the UNet. That implementation keeps the inner width at 4×4\times the channel count, so it is not parameter-matched: for 320 channels it holds 320⋅2560+1280⋅320=1,228,800320\cdot2560+1280\cdot320=1{,}228{,}800 weights against 819,200819{,}200 for a plain 4×4\times FFN, 1.5× as many.

Worked example: three hidden units, three gates

The same inputs as in the SwiGLU article: gate pre-activations g=(2.0, −1.0, 0.5)g=(2.0,\,-1.0,\,0.5), values u=(1.5, 2.0, −1.0)u=(1.5,\,2.0,\,-1.0), upstream gradient 1.

Gate ff(g)y = f(g)·u∂y/∂g
GELU (GeGLU)(1.9545, −0.1587, 0.3457)(2.9317, −0.3173, −0.3457)(1.6278, −0.1666, −0.8675)
Swish (SwiGLU)(1.7616, −0.2689, 0.3112)(2.6424, −0.5379, −0.3112)(1.6362, 0.1447, −0.7400)
ReLU (ReGLU)(2, 0, 0.5)(3, 0, −0.5)(1.5, 0, −1.0)
sigmoid (GLU)(0.8808, 0.2689, 0.6225)(1.3212, 0.5379, −0.6225)(0.1575, 0.3932, −0.2350)
Output y and gate-path gradient ∂y/∂g = u·f′(g); the value-path gradient ∂y/∂u is the gate value f(g) (4 decimals)

Unit 2 (g=−1g=-1) separates the gates. GELU's slope at −1-1 is GELU⁡′(−1)=−0.0833\operatorname{GELU}'(-1)=-0.0833, so its gate gradient is −0.167-0.167; Swish's slope is +0.0723+0.0723, giving +0.145+0.145. The two push g2g_2 in opposite directions, because −1-1 lies left of GELU's minimum (−0.75-0.75) and right of Swish's (−1.28-1.28). ReLU sends nothing at all. With the tanh approximation GeGLU's outputs become (2.9319, −0.3176, −0.3457)(2.9319,\,-0.3176,\,-0.3457).

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class GeGLUFFN(nn.Module): # T5 v1.1 naming: wi_0 gate, wi_1 value, wo out
4 def __init__(self, d, h, approximate="none"):
5 super().__init__()
6 self.wi_0 = nn.Linear(d, h, bias=False)
7 self.wi_1 = nn.Linear(d, h, bias=False)
8 self.wo = nn.Linear(h, d, bias=False)
9 self.approximate = approximate # "tanh" for Gemma-style GeGLU
10 def forward(self, x):
11 return self.wo(F.gelu(self.wi_0(x), approximate=self.approximate) * self.wi_1(x))
12
13class GEGLU(nn.Module): # diffusers-style: one fused projection
14 def __init__(self, d_in, d_out):
15 super().__init__()
16 self.proj = nn.Linear(d_in, 2 * d_out)
17 def forward(self, x):
18 value, gate = self.proj(x).chunk(2, dim=-1)
19 return value * F.gelu(gate)
20
21ffn = GeGLUFFN(768, 2048)
22print(sum(p.numel() for p in ffn.parameters())) # 4718592 = 2 * 768 * 3072
23
24g = torch.tensor([2.0, -1.0, 0.5], requires_grad=True)
25u = torch.tensor([1.5, 2.0, -1.0], requires_grad=True)
26y = F.gelu(g) * u
27y.sum().backward()
28print(y) # [ 2.9317, -0.3173, -0.3457]
29print(g.grad) # [ 1.6278, -0.1666, -0.8675]
30print(F.gelu(g, approximate="tanh") * u) # [ 2.9319, -0.3176, -0.3457]
A T5 v1.1-style GeGLU FFN, the diffusers-style fused projection, and the worked example

Use when

  • +Best log-perplexity in Shazeer's parameter-matched comparison, essentially tied with SwiGLU.
  • +Smooth gate with nonzero gradient for negative pre-activations (unlike ReGLU).
  • +Drop-in FFN replacement with the 2/3 width rule; mature fused kernels.
  • +A tanh approximation is available that is accurate to 5e-4.

Avoid when

  • −Exact GELU needs erf; the tanh form is cheaper but must match the checkpoint.
  • −Three matrices and two h-wide activations to store, like SwiGLU.
  • −Advantages over SwiGLU are within noise; the choice is mostly convention.
  • −No theoretical account of why gating with GELU helps.

Check yourself

A GeGLU unit has gate pre-activation g=−3g=-3 and value u=2u=2. Roughly how much gradient reaches uu, and how does that compare with ReGLU?

Check yourself

Which statement about GELU and β = 1 Swish as gates is correct?

Remember

  • ✓FFN⁡GEGLU(x)=(GELU⁡(xW)⊙xV)W2\operatorname{FFN}_{\text{GEGLU}}(x)=(\operatorname{GELU}(xW)\odot xV)W_2 (Shazeer 2020).
  • ✓Gradients: ∂y/∂u=GELU⁡(g)\partial y/\partial u=\operatorname{GELU}(g), ∂y/∂g=u (Φ(g)+gφ(g))\partial y/\partial g=u\,(\Phi(g)+g\varphi(g)).
  • ✓Best in Shazeer's table (1.633 at 524k steps), within noise of SwiGLU (1.636).
  • ✓GELU vs Swish: max gap 0.193 at ∣z∣≈1.97|z|\approx1.97; ReLU gating zeroes output and gradient for g<0g<0.
  • ✓Used in T5 v1.1, mT5, Gemma (tanh form) and Stable Diffusion's UNet transformer blocks (at 4× width).
← back to Deep Learning roadmap