SwiGLU

A Swish-gated GLU feed-forward layer, (Swish(xW) ⊙ xV)W₂, with hidden width cut to 8d/3 so it costs the same as a 4d ReLU FFN; the default in LLaMA-style LLMs.

9 min read668 wordsadvanced0 viewsupdated 2026-10-11

Definition

SwiGLU is the gated feed-forward block FFN⁡SwiGLU(x)=(Swish⁡(xW)⊙xV)W2\operatorname{FFN}_{\text{SwiGLU}}(x)=\big(\operatorname{Swish}(xW)\odot xV\big)W_2, a GLU whose gate uses Swish⁡(z)=z σ(z)\operatorname{Swish}(z)=z\,\sigma(z) instead of the sigmoid. It was proposed by Noam Shazeer in GLU Variants Improve Transformer (2020) and replaces the ReLU/GELU MLP inside each Transformer block.

A standard Transformer FFN expands each token from dd to 4d4d dimensions, applies a pointwise nonlinearity and projects back: φ(xW1)W2\varphi(xW_1)W_2. SwiGLU computes two expansions of the same token, passes one through Swish, multiplies them elementwise and projects back. It is less an activation function than a small change of architecture with an activation inside.

Because there are now three weight matrices instead of two, the hidden width is shrunk by 2/32/3 to keep parameters and FLOPs equal. Under that constraint SwiGLU trained to lower perplexity than ReLU or GELU, and nearly every large open language model since LLaMA uses it.

Formula

FFN⁡SwiGLU(x)=(Swish⁡1(xW)⊙xV)W2,Swish⁡β(z)=z σ(βz)\operatorname{FFN}_{\text{SwiGLU}}(x)=\Big(\operatorname{Swish}_1(xW)\odot xV\Big)W_2,\qquad \operatorname{Swish}_\beta(z)=z\,\sigma(\beta z)
x∈Rdx\in\mathbb{R}^{d}
one token's hidden state, d=dmodeld=d_{\text{model}}
W,V∈Rd×hW,V\in\mathbb{R}^{d\times h}
gate and value ("up") projections; LLaMA code calls them w1/w3 or gate_proj/up_proj
W2∈Rh×dW_2\in\mathbb{R}^{h\times d}
down projection back to dd
Swish⁡1\operatorname{Swish}_1
β=1\beta=1, also called SiLU (F.silu)
hh
hidden width, ≈23⋅4d=83d\approx\tfrac23\cdot4d=\tfrac83d to match a 4d4d standard FFN

Shazeer's FFN variants have no bias terms. Neither do LLaMA's.

Gradient

Write the gate pre-activation g=xWg=xW and the value u=xVu=xV. For each hidden unit, yk=Swish⁡(gk) uky_k=\operatorname{Swish}(g_k)\,u_k, so:

∂yk∂uk=Swish⁡(gk),∂yk∂gk=uk Swish⁡′(gk),Swish⁡′(z)=σ(z)(1+z(1−σ(z)))\frac{\partial y_k}{\partial u_k}=\operatorname{Swish}(g_k),\qquad \frac{\partial y_k}{\partial g_k}=u_k\,\operatorname{Swish}'(g_k),\qquad \operatorname{Swish}'(z)=\sigma(z)\big(1+z(1-\sigma(z))\big)
Swish⁡(gk)\operatorname{Swish}(g_k)
value-path gain; unbounded above, so an open gate can amplify as well as pass
Swish⁡′(z)\operatorname{Swish}'(z)
0.5 at 0, peaks at 1.100 near z=2.40z=2.40, slightly negative for z<−1.2785z<-1.2785 (left of the minimum), →0\to0 as z→−∞z\to-\infty

The rest is ordinary backprop through linear maps: ∂L/∂W2=y⊤δ\partial\mathcal L/\partial W_2=y^\top\delta, and the gradients reaching WW and VV are those two elementwise factors times δW2⊤\delta W_2^\top.

Matching the parameter budget

A standard FFN has 2⋅d⋅4d=8d22\cdot d\cdot4d=8d^2 weights. SwiGLU has 3dh3dh. Setting 3dh=8d23dh=8d^2 gives h=83dh=\tfrac83d. Shazeer's T5-base experiments (d=768d=768) therefore used h=2048h=2048 instead of dff=3072d_{ff}=3072: both come to 4,718,5924{,}718{,}592 weights per FFN.

LLaMA computes ⌊23⋅4d⌋\lfloor\tfrac23\cdot4d\rfloor and rounds up to a multiple of 256 for hardware-friendly shapes. For LLaMA-7B, d=4096d=4096: ⌊32768/3⌋=10922→11008\lfloor 32768/3\rfloor=10922\to\mathbf{11008}. The same rule gives 13824 for LLaMA-13B (d=5120d=5120) and 22016 for LLaMA-65B (d=8192d=8192). Later models scale the width further; LLaMA 3 8B multiplies by 1.3 and rounds to 1024, which gives 14336.

BlockHidden widthWeightsFLOPs / tokenRelative
ReLU/GELU FFN4d = 16384134,217,728268.4M1.000×
SwiGLU, LLaMA rounding11008135,266,304270.5M1.008×
SwiGLU at 4d (PaLM-style)16384201,326,592402.7M1.500×
One FFN block of LLaMA-7B (d = 4096, no biases); forward FLOPs ≈ 2 × weights per token

Across LLaMA-7B's 32 layers the SwiGLU blocks hold 32×135,266,304≈4.3332\times135{,}266{,}304\approx4.33B weights, about two thirds of the model. The LLaMA paper states it uses 23 4d\tfrac23\,4d "instead of 4d4d as in PaLM": PaLM kept the full width and paid the 1.5× cost.

standard FFN(x) = φ(xW₁)W₂x ∈ ℝ^4096W₁4096 × 16384ReLU/GELU16384 wideW₂16384 × 4096ℝ^4096SwiGLU FFN(x) = (Swish(xW) ⊙ xV)W₂x ∈ ℝ^4096W (gate)4096 × 11008V (value)4096 × 11008Swishxσ(x)⊙W₂11008 × 4096ℝ^4096weights per FFN block (no biases) · forward FLOPs per token ≈ 2 × weightsReLU/GELU FFN, 4d134M · 268M FLOPs · 1.000×SwiGLU, h = 11008135M · 271M FLOPs · 1.008×SwiGLU, h = 4d201M · 403M FLOPs · 1.500×one token, 16 of the 11008 hidden units (seed 1) · 3 with |Swish| < 0.1 are nearly shutgate xWpre-actSwish(xW)value xVproduct→ W₂Swish passes small negative gates (min −0.278 at g = −1.278), so sign and gradient survive
presets
round h up to
⌊8d/3⌋ = 10922 → h = 110083dh / 8d² = 1.0078
fig.Top: the two FFN blocks with live matrix shapes. Middle: weights and FLOPs per block; the rounded 8d/3 SwiGLU stays within 1% of the standard FFN, while a full-width SwiGLU costs 1.5×. Bottom: one token's gate pre-activations, Swish(gate), values and their product for 16 hidden units; press new token to resample. Try the presets and the rounding buttons.
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 0.731 · f′(x) = 0.928−1012345−4−2024x →Swish (β=1)Swish f′ReLUGELUSigmoid
activation
compare
range [≈ −0.278/β, ∞)monotonic nosmooth yeszero-centred no
β = 1 is SiLU. β = 0 gives x/2; β → ∞ approaches ReLU. Minimum ≈ −0.278/β at x ≈ −1.278/β.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]−0.281.823.92output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.207Var[f(x)] 0.313E[f′(x)] 0.500f′ = 0 for 0.0% of inputsE[f′]¹⁰ 9.8e−4
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.Swish, the gate nonlinearity in SwiGLU, against ReLU, GELU and the sigmoid of the original GLU.

Properties

Gate
Swish⁡(z)=zσ(z)\operatorname{Swish}(z)=z\sigma(z): smooth, non-monotone, minimum −0.2785-0.2785 at z=−1.2785z=-1.2785, unbounded above
Parameters
3dh3dh; equal to a 4d4d FFN when h=83dh=\tfrac83d
Compute
three matmuls instead of two, same total FLOPs at h=83dh=\tfrac83d; one extra elementwise multiply
Gradient
value path Swish⁡(g)\operatorname{Swish}(g), gate path u Swish⁡′(g)u\,\operatorname{Swish}'(g); nonzero for negative gates, unlike ReGLU
Paper
Shazeer, GLU Variants Improve Transformer, arXiv:2002.05202 (2020)
Used in
PaLM, LLaMA 1–3, Mistral and Mixtral, Qwen, DeepSeek, and many other decoder LLMs

What the paper found

Shazeer pre-trained T5-base-sized encoder–decoders (12+12 layers, d=768d=768) on the C4 span-filling objective, swapping only the FFN, with every variant matched for parameters and computation. Held-out log-perplexity (lower is better):

FFN65,536 steps524,288 steps
ReLU (baseline)1.9971.677
GELU1.9831.679
Swish1.9941.683
GLU (sigmoid)1.9821.663
Bilinear1.9601.648
GEGLU1.9421.633
SwiGLU1.9441.636
ReGLU1.9531.645
Table 1 of Shazeer (2020): held-out log-perplexity after 65,536 and 524,288 training steps

Every gated variant beat every ungated one, and GEGLU and SwiGLU were best, about 0.04 below the ReLU baseline after the long run. The paper reports that the GLU variants also did well on fine-tuning (GLUE, SuperGLUE, SQuAD). On why, the paper is famously candid:

Worked example: three hidden units

Gate pre-activations g=(2.0, −1.0, 0.5)g=(2.0,\,-1.0,\,0.5), values u=(1.5, 2.0, −1.0)u=(1.5,\,2.0,\,-1.0), upstream gradient 1 on each product.

UnitSwish(g)y = Swish(g)·u∂y/∂u = Swish(g)∂y/∂g = u·Swish′(g)
1: g = 2.0, u = 1.51.76162.64241.76161.6362
2: g = −1.0, u = 2.0−0.2689−0.5379−0.26890.1447
3: g = 0.5, u = −1.00.3112−0.31120.3112−0.7400
Elementwise SwiGLU core (4 decimals)

Unit 2's gate is negative, yet it outputs −0.54-0.54 and passes gradient on both paths. A ReGLU gate would output exactly 0 there and send no gradient to either g2g_2 or u2u_2. Unit 1 shows the gate amplifying: Swish⁡(2)=1.76\operatorname{Swish}(2)=1.76, so the value is scaled up, not merely passed.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3def swiglu_hidden(d, multiple_of=256):
4 h = int(2 * 4 * d / 3) # 2/3 of the usual 4d
5 return multiple_of * ((h + multiple_of - 1) // multiple_of)
6
7class SwiGLUFFN(nn.Module):
8 def __init__(self, d, multiple_of=256):
9 super().__init__()
10 h = swiglu_hidden(d, multiple_of)
11 self.w1 = nn.Linear(d, h, bias=False) # gate
12 self.w3 = nn.Linear(d, h, bias=False) # value ("up")
13 self.w2 = nn.Linear(h, d, bias=False) # down
14 def forward(self, x):
15 return self.w2(F.silu(self.w1(x)) * self.w3(x))
16
17ffn = SwiGLUFFN(4096)
18print(swiglu_hidden(4096)) # 11008
19print(sum(p.numel() for p in ffn.parameters())) # 135266304 (vs 8*4096**2 = 134217728)
20
21# the elementwise core of the worked example
22g = torch.tensor([2.0, -1.0, 0.5], requires_grad=True)
23u = torch.tensor([1.5, 2.0, -1.0], requires_grad=True)
24y = F.silu(g) * u
25y.sum().backward()
26print(y) # [ 2.6424, -0.5379, -0.3112]
27print(g.grad) # [ 1.6362, 0.1447, -0.7400]
28print(u.grad) # [ 1.7616, -0.2689, 0.3112]
A LLaMA-style SwiGLU FFN with the 8d/3 sizing rule

Use when

  • +Lower perplexity than ReLU/GELU FFNs at matched parameters and FLOPs (Shazeer 2020), confirmed in practice by its adoption across LLMs.
  • +Smooth gate with nonzero gradient for negative inputs; no dead units.
  • +Drop-in replacement for the FFN; the sizing rule keeps parameter budgets unchanged.
  • +Well supported: fused SiLU-and-multiply kernels exist in most inference stacks.

Avoid when

  • −Three matmuls of awkward width (8d/3) instead of two; rounding is needed for efficient tiling.
  • −More activation memory per token during training.
  • −Swish costs an exponential per element (minor next to the matmuls).
  • −No theoretical justification; benefits are empirical and modest per layer.

Check yourself

A model has dmodel=5120d_{\text{model}}=5120. Using LLaMA's rule (⌊8d/3⌋ rounded up to a multiple of 256), what is the SwiGLU hidden width?

Check yourself

Why is the SwiGLU hidden width about 2/3 of the standard FFN's 4d?

Remember

  • ✓FFN⁡SwiGLU(x)=(Swish⁡(xW)⊙xV)W2\operatorname{FFN}_{\text{SwiGLU}}(x)=(\operatorname{Swish}(xW)\odot xV)W_2, no biases (Shazeer 2020).
  • ✓Gradients: ∂y/∂u=Swish⁡(g)\partial y/\partial u=\operatorname{Swish}(g), ∂y/∂g=u Swish⁡′(g)\partial y/\partial g=u\,\operatorname{Swish}'(g).
  • ✓Hidden width 83d\tfrac83d matches a 4d4d FFN: 3dh=8d23dh=8d^2. LLaMA-7B: d=4096→11008d=4096\to11008.
  • ✓Shazeer's C4 results: SwiGLU 1.636 vs ReLU 1.677 log-perplexity after 524k steps; GEGLU 1.633.
  • ✓Used in PaLM, LLaMA 1–3, Mistral, Qwen and most open LLMs; nobody has a full explanation of why it helps.
← back to Deep Learning roadmap