Definition
SwiGLU is the gated feed-forward block , a GLU whose gate uses instead of the sigmoid. It was proposed by Noam Shazeer in GLU Variants Improve Transformer (2020) and replaces the ReLU/GELU MLP inside each Transformer block.
A standard Transformer FFN expands each token from to dimensions, applies a pointwise nonlinearity and projects back: . SwiGLU computes two expansions of the same token, passes one through Swish, multiplies them elementwise and projects back. It is less an activation function than a small change of architecture with an activation inside.
Because there are now three weight matrices instead of two, the hidden width is shrunk by to keep parameters and FLOPs equal. Under that constraint SwiGLU trained to lower perplexity than ReLU or GELU, and nearly every large open language model since LLaMA uses it.
Formula
- one token's hidden state,
- gate and value ("up") projections; LLaMA code calls them
w1/w3orgate_proj/up_proj - down projection back to
- , also called SiLU (
F.silu) - hidden width, to match a standard FFN
Shazeer's FFN variants have no bias terms. Neither do LLaMA's.
Gradient
Write the gate pre-activation and the value . For each hidden unit, , so:
- value-path gain; unbounded above, so an open gate can amplify as well as pass
- 0.5 at 0, peaks at 1.100 near , slightly negative for (left of the minimum), as
The rest is ordinary backprop through linear maps: , and the gradients reaching and are those two elementwise factors times .
Matching the parameter budget
A standard FFN has weights. SwiGLU has . Setting gives . Shazeer's T5-base experiments () therefore used instead of : both come to weights per FFN.
LLaMA computes and rounds up to a multiple of 256 for hardware-friendly shapes. For LLaMA-7B, : . The same rule gives 13824 for LLaMA-13B () and 22016 for LLaMA-65B (). Later models scale the width further; LLaMA 3 8B multiplies by 1.3 and rounds to 1024, which gives 14336.
| Block | Hidden width | Weights | FLOPs / token | Relative |
|---|---|---|---|---|
| ReLU/GELU FFN | 4d = 16384 | 134,217,728 | 268.4M | 1.000× |
| SwiGLU, LLaMA rounding | 11008 | 135,266,304 | 270.5M | 1.008× |
| SwiGLU at 4d (PaLM-style) | 16384 | 201,326,592 | 402.7M | 1.500× |
Across LLaMA-7B's 32 layers the SwiGLU blocks hold B weights, about two thirds of the model. The LLaMA paper states it uses "instead of as in PaLM": PaLM kept the full width and paid the 1.5× cost.
Properties
- Gate
- : smooth, non-monotone, minimum at , unbounded above
- Parameters
- ; equal to a FFN when
- Compute
- three matmuls instead of two, same total FLOPs at ; one extra elementwise multiply
- Gradient
- value path , gate path ; nonzero for negative gates, unlike ReGLU
- Paper
- Shazeer, GLU Variants Improve Transformer, arXiv:2002.05202 (2020)
- Used in
- PaLM, LLaMA 1–3, Mistral and Mixtral, Qwen, DeepSeek, and many other decoder LLMs
What the paper found
Shazeer pre-trained T5-base-sized encoder–decoders (12+12 layers, ) on the C4 span-filling objective, swapping only the FFN, with every variant matched for parameters and computation. Held-out log-perplexity (lower is better):
| FFN | 65,536 steps | 524,288 steps |
|---|---|---|
| ReLU (baseline) | 1.997 | 1.677 |
| GELU | 1.983 | 1.679 |
| Swish | 1.994 | 1.683 |
| GLU (sigmoid) | 1.982 | 1.663 |
| Bilinear | 1.960 | 1.648 |
| GEGLU | 1.942 | 1.633 |
| SwiGLU | 1.944 | 1.636 |
| ReGLU | 1.953 | 1.645 |
Every gated variant beat every ungated one, and GEGLU and SwiGLU were best, about 0.04 below the ReLU baseline after the long run. The paper reports that the GLU variants also did well on fine-tuning (GLUE, SuperGLUE, SQuAD). On why, the paper is famously candid:
Worked example: three hidden units
Gate pre-activations , values , upstream gradient 1 on each product.
| Unit | Swish(g) | y = Swish(g)·u | ∂y/∂u = Swish(g) | ∂y/∂g = u·Swish′(g) |
|---|---|---|---|---|
| 1: g = 2.0, u = 1.5 | 1.7616 | 2.6424 | 1.7616 | 1.6362 |
| 2: g = −1.0, u = 2.0 | −0.2689 | −0.5379 | −0.2689 | 0.1447 |
| 3: g = 0.5, u = −1.0 | 0.3112 | −0.3112 | 0.3112 | −0.7400 |
Unit 2's gate is negative, yet it outputs and passes gradient on both paths. A ReGLU gate would output exactly 0 there and send no gradient to either or . Unit 1 shows the gate amplifying: , so the value is scaled up, not merely passed.
1import torch, torch.nn as nn, torch.nn.functional as F23def swiglu_hidden(d, multiple_of=256):4 h = int(2 * 4 * d / 3) # 2/3 of the usual 4d5 return multiple_of * ((h + multiple_of - 1) // multiple_of)67class SwiGLUFFN(nn.Module):8 def __init__(self, d, multiple_of=256):9 super().__init__()10 h = swiglu_hidden(d, multiple_of)11 self.w1 = nn.Linear(d, h, bias=False) # gate12 self.w3 = nn.Linear(d, h, bias=False) # value ("up")13 self.w2 = nn.Linear(h, d, bias=False) # down14 def forward(self, x):15 return self.w2(F.silu(self.w1(x)) * self.w3(x))1617ffn = SwiGLUFFN(4096)18print(swiglu_hidden(4096)) # 1100819print(sum(p.numel() for p in ffn.parameters())) # 135266304 (vs 8*4096**2 = 134217728)2021# the elementwise core of the worked example22g = torch.tensor([2.0, -1.0, 0.5], requires_grad=True)23u = torch.tensor([1.5, 2.0, -1.0], requires_grad=True)24y = F.silu(g) * u25y.sum().backward()26print(y) # [ 2.6424, -0.5379, -0.3112]27print(g.grad) # [ 1.6362, 0.1447, -0.7400]28print(u.grad) # [ 1.7616, -0.2689, 0.3112]Use when
- +Lower perplexity than ReLU/GELU FFNs at matched parameters and FLOPs (Shazeer 2020), confirmed in practice by its adoption across LLMs.
- +Smooth gate with nonzero gradient for negative inputs; no dead units.
- +Drop-in replacement for the FFN; the sizing rule keeps parameter budgets unchanged.
- +Well supported: fused SiLU-and-multiply kernels exist in most inference stacks.
Avoid when
- −Three matmuls of awkward width (8d/3) instead of two; rounding is needed for efficient tiling.
- −More activation memory per token during training.
- −Swish costs an exponential per element (minor next to the matmuls).
- −No theoretical justification; benefits are empirical and modest per layer.
Check yourself
A model has . Using LLaMA's rule (⌊8d/3⌋ rounded up to a multiple of 256), what is the SwiGLU hidden width?
Check yourself
Why is the SwiGLU hidden width about 2/3 of the standard FFN's 4d?
Remember
- ✓, no biases (Shazeer 2020).
- ✓Gradients: , .
- ✓Hidden width matches a FFN: . LLaMA-7B: .
- ✓Shazeer's C4 results: SwiGLU 1.636 vs ReLU 1.677 log-perplexity after 524k steps; GEGLU 1.633.
- ✓Used in PaLM, LLaMA 1–3, Mistral, Qwen and most open LLMs; nobody has a full explanation of why it helps.