# SwiGLU

> A Swish-gated GLU feed-forward layer, (Swish(xW) ⊙ xV)W₂, with hidden width cut to 8d/3 so it costs the same as a 4d ReLU FFN; the default in LLaMA-style LLMs.

Source: https://www.mlfoundry.in/learn/swiglu  
Section: Deep Learning › Activation Functions  
Level: advanced · 9 min read · updated 2026-10-11

**SwiGLU:** is the gated feed-forward block $\operatorname{FFN}_{\text{SwiGLU}}(x)=\big(\operatorname{Swish}(xW)\odot xV\big)W_2$, a GLU whose gate uses $\operatorname{Swish}(z)=z\,\sigma(z)$ instead of the sigmoid. It was proposed by Noam Shazeer in *GLU Variants Improve Transformer* (2020) and replaces the ReLU/GELU MLP inside each Transformer block.

A standard Transformer FFN expands each token from $d$ to $4d$ dimensions, applies a pointwise nonlinearity and projects back: $\varphi(xW_1)W_2$. SwiGLU computes two expansions of the same token, passes one through Swish, multiplies them elementwise and projects back. It is less an activation function than a small change of architecture with an activation inside.

Because there are now three weight matrices instead of two, the hidden width is shrunk by $2/3$ to keep parameters and FLOPs equal. Under that constraint SwiGLU trained to lower perplexity than ReLU or GELU, and nearly every large open language model since LLaMA uses it.

## Formula

$$
\operatorname{FFN}_{\text{SwiGLU}}(x)=\Big(\operatorname{Swish}_1(xW)\odot xV\Big)W_2,\qquad \operatorname{Swish}_\beta(z)=z\,\sigma(\beta z)
$$

- $x\in\mathbb{R}^{d}$: one token's hidden state, $d=d_{\text{model}}$
- $W,V\in\mathbb{R}^{d\times h}$: gate and value ("up") projections; LLaMA code calls them `w1`/`w3` or `gate_proj`/`up_proj`
- $W_2\in\mathbb{R}^{h\times d}$: down projection back to $d$
- $\operatorname{Swish}_1$: $\beta=1$, also called SiLU (`F.silu`)
- $h$: hidden width, $\approx\tfrac23\cdot4d=\tfrac83d$ to match a $4d$ standard FFN

Shazeer's FFN variants have no bias terms. Neither do LLaMA's.

## Gradient

Write the gate pre-activation $g=xW$ and the value $u=xV$. For each hidden unit, $y_k=\operatorname{Swish}(g_k)\,u_k$, so:

$$
\frac{\partial y_k}{\partial u_k}=\operatorname{Swish}(g_k),\qquad \frac{\partial y_k}{\partial g_k}=u_k\,\operatorname{Swish}'(g_k),\qquad \operatorname{Swish}'(z)=\sigma(z)\big(1+z(1-\sigma(z))\big)
$$

- $\operatorname{Swish}(g_k)$: value-path gain; unbounded above, so an open gate can amplify as well as pass
- $\operatorname{Swish}'(z)$: 0.5 at 0, peaks at 1.100 near $z=2.40$, slightly negative for $z<-1.2785$ (left of the minimum), $\to0$ as $z\to-\infty$

The rest is ordinary backprop through linear maps: $\partial\mathcal L/\partial W_2=y^\top\delta$, and the gradients reaching $W$ and $V$ are those two elementwise factors times $\delta W_2^\top$.

## Matching the parameter budget

A standard FFN has $2\cdot d\cdot4d=8d^2$ weights. SwiGLU has $3dh$. Setting $3dh=8d^2$ gives $h=\tfrac83d$. Shazeer's T5-base experiments ($d=768$) therefore used $h=2048$ instead of $d_{ff}=3072$: both come to $4{,}718{,}592$ weights per FFN.

LLaMA computes $\lfloor\tfrac23\cdot4d\rfloor$ and rounds **up** to a multiple of 256 for hardware-friendly shapes. For LLaMA-7B, $d=4096$: $\lfloor 32768/3\rfloor=10922\to\mathbf{11008}$. The same rule gives 13824 for LLaMA-13B ($d=5120$) and 22016 for LLaMA-65B ($d=8192$). Later models scale the width further; LLaMA 3 8B multiplies by 1.3 and rounds to 1024, which gives 14336.

One FFN block of LLaMA-7B (d = 4096, no biases); forward FLOPs ≈ 2 × weights per token

| Block | Hidden width | Weights | FLOPs / token | Relative |
| --- | --- | --- | --- | --- |
| ReLU/GELU FFN | 4d = 16384 | 134,217,728 | 268.4M | 1.000× |
| SwiGLU, LLaMA rounding | 11008 | 135,266,304 | 270.5M | 1.008× |
| SwiGLU at 4d (PaLM-style) | 16384 | 201,326,592 | 402.7M | 1.500× |

Across LLaMA-7B's 32 layers the SwiGLU blocks hold $32\times135{,}266{,}304\approx4.33$B weights, about two thirds of the model. The LLaMA paper states it uses $\tfrac23\,4d$ "instead of $4d$ as in PaLM": PaLM kept the full width and paid the 1.5× cost.

> Figure: Top: the two FFN blocks with live matrix shapes. Middle: weights and FLOPs per block; the rounded 8d/3 SwiGLU stays within 1% of the standard FFN, while a full-width SwiGLU costs 1.5×. Bottom: one token's gate pre-activations, Swish(gate), values and their product for 16 hidden units; press new token to resample. Try the presets and the rounding buttons.

> Figure: Swish, the gate nonlinearity in SwiGLU, against ReLU, GELU and the sigmoid of the original GLU.

**Properties**

- **Gate:** $\operatorname{Swish}(z)=z\sigma(z)$: smooth, non-monotone, minimum $-0.2785$ at $z=-1.2785$, unbounded above
- **Parameters:** $3dh$; equal to a $4d$ FFN when $h=\tfrac83d$
- **Compute:** three matmuls instead of two, same total FLOPs at $h=\tfrac83d$; one extra elementwise multiply
- **Gradient:** value path $\operatorname{Swish}(g)$, gate path $u\,\operatorname{Swish}'(g)$; nonzero for negative gates, unlike ReGLU
- **Paper:** Shazeer, *GLU Variants Improve Transformer*, arXiv:2002.05202 (2020)
- **Used in:** PaLM, LLaMA 1–3, Mistral and Mixtral, Qwen, DeepSeek, and many other decoder LLMs

## What the paper found

Shazeer pre-trained T5-base-sized encoder–decoders (12+12 layers, $d=768$) on the C4 span-filling objective, swapping only the FFN, with every variant matched for parameters and computation. Held-out log-perplexity (lower is better):

Table 1 of Shazeer (2020): held-out log-perplexity after 65,536 and 524,288 training steps

| FFN | 65,536 steps | 524,288 steps |
| --- | --- | --- |
| ReLU (baseline) | 1.997 | 1.677 |
| GELU | 1.983 | 1.679 |
| Swish | 1.994 | 1.683 |
| GLU (sigmoid) | 1.982 | 1.663 |
| Bilinear | 1.960 | 1.648 |
| GEGLU | 1.942 | 1.633 |
| SwiGLU | 1.944 | 1.636 |
| ReGLU | 1.953 | 1.645 |

Every gated variant beat every ungated one, and GEGLU and SwiGLU were best, about 0.04 below the ReLU baseline after the long run. The paper reports that the GLU variants also did well on fine-tuning (GLUE, SuperGLUE, SQuAD). On why, the paper is famously candid:

> **Shazeer (2020), conclusion:** "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence."

## Worked example: three hidden units

Gate pre-activations $g=(2.0,\,-1.0,\,0.5)$, values $u=(1.5,\,2.0,\,-1.0)$, upstream gradient 1 on each product.

Elementwise SwiGLU core (4 decimals)

| Unit | Swish(g) | y = Swish(g)·u | ∂y/∂u = Swish(g) | ∂y/∂g = u·Swish′(g) |
| --- | --- | --- | --- | --- |
| 1: g = 2.0, u = 1.5 | 1.7616 | 2.6424 | 1.7616 | 1.6362 |
| 2: g = −1.0, u = 2.0 | −0.2689 | −0.5379 | −0.2689 | 0.1447 |
| 3: g = 0.5, u = −1.0 | 0.3112 | −0.3112 | 0.3112 | −0.7400 |

Unit 2's gate is negative, yet it outputs $-0.54$ and passes gradient on both paths. A ReGLU gate would output exactly 0 there and send no gradient to either $g_2$ or $u_2$. Unit 1 shows the gate amplifying: $\operatorname{Swish}(2)=1.76$, so the value is scaled up, not merely passed.

```python
import torch, torch.nn as nn, torch.nn.functional as F

def swiglu_hidden(d, multiple_of=256):
    h = int(2 * 4 * d / 3)                    # 2/3 of the usual 4d
    return multiple_of * ((h + multiple_of - 1) // multiple_of)

class SwiGLUFFN(nn.Module):
    def __init__(self, d, multiple_of=256):
        super().__init__()
        h = swiglu_hidden(d, multiple_of)
        self.w1 = nn.Linear(d, h, bias=False)   # gate
        self.w3 = nn.Linear(d, h, bias=False)   # value ("up")
        self.w2 = nn.Linear(h, d, bias=False)   # down
    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

ffn = SwiGLUFFN(4096)
print(swiglu_hidden(4096))                     # 11008
print(sum(p.numel() for p in ffn.parameters()))  # 135266304 (vs 8*4096**2 = 134217728)

# the elementwise core of the worked example
g = torch.tensor([2.0, -1.0, 0.5], requires_grad=True)
u = torch.tensor([1.5, 2.0, -1.0], requires_grad=True)
y = F.silu(g) * u
y.sum().backward()
print(y)        # [ 2.6424, -0.5379, -0.3112]
print(g.grad)   # [ 1.6362,  0.1447, -0.7400]
print(u.grad)   # [ 1.7616, -0.2689,  0.3112]
```

A LLaMA-style SwiGLU FFN with the 8d/3 sizing rule

> **Do not keep the 4d width by accident:** Swapping `nn.Sequential(Linear(d, 4d), GELU(), Linear(4d, d))` for a SwiGLU block with `h = 4d` makes every FFN 1.5× bigger and slower. That is a legitimate choice (PaLM made it), but then compare against a baseline with the same parameter count, or the gain is mostly extra capacity.

> **Activation memory:** Training must keep both $h$-wide tensors $xW$ and $xV$ (or recompute them) for the backward pass, versus one $4d$-wide tensor for a standard FFN. At $h=\tfrac83d$ that is $\tfrac{16}3d$ values per token instead of $4d$, unless the kernel is fused or checkpointed.

> **Why gating might help:** The paper offers no explanation, but the mechanics are visible: each hidden unit is a product of two learned projections, a quadratic interaction the plain FFN cannot express in one layer, and Swish keeps a nonzero gradient for slightly negative gates. Treat these as intuitions, not proofs; the evidence is empirical and has been replicated at scale.

**Strengths**

- Lower perplexity than ReLU/GELU FFNs at matched parameters and FLOPs (Shazeer 2020), confirmed in practice by its adoption across LLMs.
- Smooth gate with nonzero gradient for negative inputs; no dead units.
- Drop-in replacement for the FFN; the sizing rule keeps parameter budgets unchanged.
- Well supported: fused SiLU-and-multiply kernels exist in most inference stacks.

**Limitations**

- Three matmuls of awkward width (8d/3) instead of two; rounding is needed for efficient tiling.
- More activation memory per token during training.
- Swish costs an exponential per element (minor next to the matmuls).
- No theoretical justification; benefits are empirical and modest per layer.

**Question:** A model has $d_{\text{model}}=5120$. Using LLaMA's rule (⌊8d/3⌋ rounded up to a multiple of 256), what is the SwiGLU hidden width?

- A. 13653
- B. 13824
- C. 20480
- D. 13568

**Answer:** B. 13824 — $8\cdot5120/3=13653.3$, floor $13653$. The next multiple of 256 is $54\cdot256=13824$ (LLaMA-13B's value). 20480 is $4d$; 13568 rounds down instead of up.

**Question:** Why is the SwiGLU hidden width about 2/3 of the standard FFN's 4d?

- A. Swish has a smaller output range than ReLU
- B. So three d×h matrices hold the same number of weights as two d×4d matrices
- C. To make the hidden width a power of two
- D. Because the gate halves the number of active units

**Answer:** B. So three d×h matrices hold the same number of weights as two d×4d matrices — $3dh=2\cdot d\cdot4d\Rightarrow h=\tfrac83d=\tfrac23\cdot4d$. Shazeer matched parameters and computation this way, and LLaMA adopted the same budget.

**Key points**

- $\operatorname{FFN}_{\text{SwiGLU}}(x)=(\operatorname{Swish}(xW)\odot xV)W_2$, no biases (Shazeer 2020).
- Gradients: $\partial y/\partial u=\operatorname{Swish}(g)$, $\partial y/\partial g=u\,\operatorname{Swish}'(g)$.
- Hidden width $\tfrac83d$ matches a $4d$ FFN: $3dh=8d^2$. LLaMA-7B: $d=4096\to11008$.
- Shazeer's C4 results: SwiGLU 1.636 vs ReLU 1.677 log-perplexity after 524k steps; GEGLU 1.633.
- Used in PaLM, LLaMA 1–3, Mistral, Qwen and most open LLMs; nobody has a full explanation of why it helps.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
