Definition
GeGLU (also written GEGLU) is the gated feed-forward block : a GLU whose gate is the Gaussian Error Linear Unit . Like SwiGLU it comes from Shazeer, GLU Variants Improve Transformer (2020).
GeGLU and SwiGLU are siblings: same three-matrix block, same hidden-width rule, different gate curve. In Shazeer's experiments the two finished essentially tied at the top. Model families tend to pick one and stick with it: Google's T5 v1.1 and Gemma use GeGLU, Meta's LLaMA line uses SwiGLU.
This article focuses on what the gate function changes inside the product: the shape of the output, and how gradient is split between the gate and value paths.
Formula
- gate and value projections; T5 v1.1 calls them
wi_0andwi_1 - output projection (
wo) - standard normal CDF
- of a standard FFN's width at equal parameters, e.g. 2048 instead of 3072 for
- the tanh approximation (Hendrycks & Gimpel 2016),
F.gelu(x, approximate="tanh"); within of the exact GELU everywhere (worst near ). Gemma's "approximated GeGLU" uses it
Gradient
With gate pre-activation and value , each hidden unit computes :
- standard normal density
- 0.5 at 0, peak 1.1289 at , negative for (left of GELU's minimum )
Properties
- Gate
- : smooth, non-monotone, minimum at , unbounded above
- vs Swish
- at ; vs the gap is only
- Parameters
- , same as SwiGLU; equal to a FFN at
- Gradient
- value path , gate path ; both nonzero for negative
- Shazeer 2020
- log-perplexity 1.942 / 1.633 (65k / 524k steps), the best of the eight FFNs compared; SwiGLU 1.944 / 1.636, ReLU 1.997 / 1.677
- Used in
- T5 v1.1 and mT5, Gemma (tanh-approximated), the transformer blocks of Stable Diffusion's UNet (
GEGLUin diffusers and the original latent-diffusion code)
GELU vs Swish vs ReLU inside the gate
GELU and Swish look alike, but Swish is a noticeably softer curve: the largest gap is at (at : GELU , Swish ). Their derivatives differ by up to near . GELU is much closer to , i.e. Swish with (max gap ). Shazeer's results put the two gates within noise of each other, so the choice matters less than the gating itself.
ReLU (ReGLU) is the hard version: for the gate is exactly zero, so the output and both gradients vanish for that unit and that token. Sigmoid (the original GLU) never closes fully but also never exceeds 1, so it can only attenuate; GELU and Swish can amplify a value by more than its own size when the gate is large.
History and use
GELU was proposed by Hendrycks and Gimpel (2016) as a smooth, probabilistic ReLU and became the FFN activation of BERT and GPT-2. Shazeer (2020) used it as a gate and got the best numbers of his comparison. Google's T5 v1.1 checkpoints replaced ReLU with GeGLU in the FFN (with for the base size), and mT5 follows that recipe. Gemma (2024) uses the tanh-approximated GeGLU.
Latent diffusion / Stable Diffusion put a GeGLU FFN in every spatial transformer block of the UNet. That implementation keeps the inner width at the channel count, so it is not parameter-matched: for 320 channels it holds weights against for a plain FFN, 1.5× as many.
Worked example: three hidden units, three gates
The same inputs as in the SwiGLU article: gate pre-activations , values , upstream gradient 1.
| Gate f | f(g) | y = f(g)·u | ∂y/∂g |
|---|---|---|---|
| GELU (GeGLU) | (1.9545, −0.1587, 0.3457) | (2.9317, −0.3173, −0.3457) | (1.6278, −0.1666, −0.8675) |
| Swish (SwiGLU) | (1.7616, −0.2689, 0.3112) | (2.6424, −0.5379, −0.3112) | (1.6362, 0.1447, −0.7400) |
| ReLU (ReGLU) | (2, 0, 0.5) | (3, 0, −0.5) | (1.5, 0, −1.0) |
| sigmoid (GLU) | (0.8808, 0.2689, 0.6225) | (1.3212, 0.5379, −0.6225) | (0.1575, 0.3932, −0.2350) |
Unit 2 () separates the gates. GELU's slope at is , so its gate gradient is ; Swish's slope is , giving . The two push in opposite directions, because lies left of GELU's minimum () and right of Swish's (). ReLU sends nothing at all. With the tanh approximation GeGLU's outputs become .
1import torch, torch.nn as nn, torch.nn.functional as F23class GeGLUFFN(nn.Module): # T5 v1.1 naming: wi_0 gate, wi_1 value, wo out4 def __init__(self, d, h, approximate="none"):5 super().__init__()6 self.wi_0 = nn.Linear(d, h, bias=False)7 self.wi_1 = nn.Linear(d, h, bias=False)8 self.wo = nn.Linear(h, d, bias=False)9 self.approximate = approximate # "tanh" for Gemma-style GeGLU10 def forward(self, x):11 return self.wo(F.gelu(self.wi_0(x), approximate=self.approximate) * self.wi_1(x))1213class GEGLU(nn.Module): # diffusers-style: one fused projection14 def __init__(self, d_in, d_out):15 super().__init__()16 self.proj = nn.Linear(d_in, 2 * d_out)17 def forward(self, x):18 value, gate = self.proj(x).chunk(2, dim=-1)19 return value * F.gelu(gate)2021ffn = GeGLUFFN(768, 2048)22print(sum(p.numel() for p in ffn.parameters())) # 4718592 = 2 * 768 * 30722324g = torch.tensor([2.0, -1.0, 0.5], requires_grad=True)25u = torch.tensor([1.5, 2.0, -1.0], requires_grad=True)26y = F.gelu(g) * u27y.sum().backward()28print(y) # [ 2.9317, -0.3173, -0.3457]29print(g.grad) # [ 1.6278, -0.1666, -0.8675]30print(F.gelu(g, approximate="tanh") * u) # [ 2.9319, -0.3176, -0.3457]Use when
- +Best log-perplexity in Shazeer's parameter-matched comparison, essentially tied with SwiGLU.
- +Smooth gate with nonzero gradient for negative pre-activations (unlike ReGLU).
- +Drop-in FFN replacement with the 2/3 width rule; mature fused kernels.
- +A tanh approximation is available that is accurate to 5e-4.
Avoid when
- −Exact GELU needs erf; the tanh form is cheaper but must match the checkpoint.
- −Three matrices and two h-wide activations to store, like SwiGLU.
- −Advantages over SwiGLU are within noise; the choice is mostly convention.
- −No theoretical account of why gating with GELU helps.
Check yourself
A GeGLU unit has gate pre-activation and value . Roughly how much gradient reaches , and how does that compare with ReGLU?
Check yourself
Which statement about GELU and β = 1 Swish as gates is correct?
Remember
- ✓ (Shazeer 2020).
- ✓Gradients: , .
- ✓Best in Shazeer's table (1.633 at 524k steps), within noise of SwiGLU (1.636).
- ✓GELU vs Swish: max gap 0.193 at ; ReLU gating zeroes output and gradient for .
- ✓Used in T5 v1.1, mT5, Gemma (tanh form) and Stable Diffusion's UNet transformer blocks (at 4× width).