# GELU

> Weight each input by the probability that a standard normal falls below it: x·Φ(x), a smooth ReLU and the default activation of transformers.

Source: https://www.mlfoundry.in/learn/gelu  
Section: Deep Learning › Activation Functions  
Level: core · 8 min read · updated 2026-10-11

**GELU (Gaussian Error Linear Unit):** is the activation $\mathrm{GELU}(x)=x\,\Phi(x)$, where $\Phi$ is the standard normal CDF. It scales each input by how likely a standard normal variable is to be smaller than it, instead of gating it by its sign as ReLU does.

ReLU multiplies its input by a hard 0/1 gate, $\mathbb{1}[x>0]$. GELU swaps the hard gate for a soft one, $\Phi(x)\in(0,1)$. Large positive inputs pass almost unchanged and large negative inputs are almost zeroed, as with ReLU. Near zero the gate is partial, which makes the curve smooth and lets it dip slightly below zero.

Hendrycks & Gimpel proposed it in 2016 ("Gaussian Error Linear Units"). It became the standard feed-forward activation of transformers: BERT, GPT-2, GPT-3 and the Vision Transformer (ViT) all use it.

## Formula and derivative

$$
\mathrm{GELU}(x)=x\,\Phi(x)=\frac{x}{2}\Big(1+\operatorname{erf}\big(x/\sqrt2\big)\Big)
$$

- $\Phi(x)$: standard normal CDF, $P(Z\le x)$ for $Z\sim\mathcal N(0,1)$
- $\operatorname{erf}$: the error function, $\Phi(x)=\tfrac12\big(1+\operatorname{erf}(x/\sqrt2)\big)$

The product rule gives the derivative. The second term is what lets the slope go negative and rise slightly above 1:

$$
\mathrm{GELU}'(x)=\Phi(x)+x\,\varphi(x),\qquad \mathrm{GELU}''(x)=\varphi(x)\,(2-x^2)
$$

- $\varphi(x)$: standard normal density, $e^{-x^2/2}/\sqrt{2\pi}$ (note $\Phi'=\varphi$ and $\varphi'(x)=-x\varphi(x)$)
- $\Phi(x)$: the gate itself: the slope ReLU would have if its gate were frozen
- $x\varphi(x)$: the extra term from the gate changing with x; negative for x < 0

Setting $\mathrm{GELU}'(x)=0$ and solving numerically gives the dip: the minimum is $-0.1700$ at $x=-0.7518$. The slope peaks at $1.1289$ at $x=\sqrt2$ and is lowest, $-0.1289$, at $x=-\sqrt2$, where $\mathrm{GELU}''=0$. Since $\mathrm{GELU}(x)-\mathrm{GELU}(-x)=x$ (because $\Phi(-x)=1-\Phi(x)$), the slopes at $\pm x$ always sum to 1.

**At a glance**

- **Range:** $[-0.170,\ \infty)$; unbounded above, bounded below
- **Monotonic:** No. Decreasing on $(-\infty,-0.752)$, increasing after
- **Smooth:** Yes, infinitely differentiable; $\mathrm{GELU}'(0)=0.5$, $\mathrm{GELU}''(0)=\sqrt{2/\pi}\approx0.798$
- **Zero-centred:** No. For $x\sim\mathcal N(0,1)$ the output has mean $1/(2\sqrt\pi)\approx0.282$ and std $0.588$
- **Saturation:** Only on the negative side ($\to0$ as $x\to-\infty$); no saturation for positive inputs
- **Cost:** One erf (or one tanh in the approximation) per element: more than ReLU's compare, though usually fused into the matmul epilogue

## Where it comes from: an expected stochastic gate

Hendrycks & Gimpel derived GELU from a random regulariser. Multiply the input by a mask $m\sim\mathrm{Bernoulli}(\Phi(x))$, so an input is kept with probability $\Phi(x)$. Small inputs are usually dropped, large ones usually kept. The deterministic activation is the expected output:

$$
\mathbb{E}[\,x\cdot m\,]=x\cdot P(m=1)+0\cdot P(m=0)=x\,\Phi(x)
$$

- $m$: random keep-mask, $m=1$ with probability $\Phi(x)$
- $\Phi(x)$: keep probability: since pre-activations are often roughly standard normal (e.g. after normalization), it is the chance that a typical input is smaller than this one

This combines two existing ideas. **Dropout** multiplies by a random mask whose keep probability is fixed. **Zoneout** (for RNNs) randomly keeps the previous hidden state. ReLU applies a deterministic mask, $\mathbb{1}[x>0]$. GELU's mask is random but depends on the input, and the network uses its average.

> Figure: Each draw samples one Bernoulli(Φ(x)) mask per probe: the output is either x (orange dot on the dashed identity line) or 0 (hollow dot). Blue squares are running means over all draws. Press play: the means settle onto x·Φ(x) at the 1/√n rate, and the zoom on the right shows them finding the dip near −0.75. The bottom panels plot the error of the two closed-form approximations.

> Figure: GELU against ReLU and Swish. Move the probe into the negative region: ReLU's slope is exactly 0 there, while GELU still passes a small (sometimes negative) gradient.

## The two approximations

An erf was not always a fast primitive, so the paper also gives two cheaper forms:

$$
\mathrm{GELU}(x)\approx\tfrac12x\Big(1+\tanh\!\Big[\sqrt{2/\pi}\,\big(x+0.044715\,x^3\big)\Big]\Big)\approx x\,\sigma(1.702\,x)
$$

- $\tanh[\cdot]$: tanh approximation; max absolute error $4.7\times10^{-4}$ (at $x\approx\pm2.70$)
- $\sigma(1.702x)$: sigmoid approximation, i.e. Swish with $\beta=1.702$; max absolute error $0.020$ (at $x\approx\pm2.27$)

The original BERT and GPT-2 code used the tanh form, which is why libraries keep it: PyTorch's `approximate="tanh"` and Hugging Face's `gelu_new` exist to reproduce those checkpoints exactly. On GPUs the elementwise op is memory-bound, so for a new model the exact erf form typically costs about the same.

## Worked example

GELU, its approximations and its slope at a few inputs (4 decimals; the highlighted row is the minimum)

| x | Φ(x) | GELU(x) | tanh approx | x·σ(1.702x) | GELU′(x) |
| --- | --- | --- | --- | --- | --- |
| −3 | 0.0013 | −0.0040 | −0.0036 | −0.0181 | −0.0119 |
| −2 | 0.0228 | −0.0455 | −0.0454 | −0.0643 | −0.0852 |
| −1 | 0.1587 | −0.1587 | −0.1588 | −0.1542 | −0.0833 |
| −0.7518 | 0.2261 | −0.1700 | −0.1700 | −0.1636 | 0.0000 |
| −0.5 | 0.3085 | −0.1543 | −0.1543 | −0.1496 | 0.1325 |
| 0 | 0.5000 | 0.0000 | 0.0000 | 0.0000 | 0.5000 |
| 0.5 | 0.6915 | 0.3457 | 0.3457 | 0.3504 | 0.8675 |
| 1 | 0.8413 | 0.8413 | 0.8412 | 0.8458 | 1.0833 |
| 2 | 0.9772 | 1.9545 | 1.9546 | 1.9357 | 1.0852 |

Two readings. At $x=-1$, GELU equals $-\Phi(-1)=-0.1587$, which is just $x\Phi(x)$. And for any $x$ between $-0.75$ and $0$ the slope is positive but well below 1. At $x=-0.5$ a ReLU unit sends back no gradient, while a GELU unit sends back $0.1325$ times the upstream gradient.

## Effect on training

**Gradient flow.** The slope is never exactly zero on an interval, so GELU has no "dead" units in ReLU's sense. Negative pre-activations still receive small gradients. For $x<-0.752$ the slope is negative, so the gradient reaching the input has the opposite sign to the upstream one, but it is tiny (the slope is never below $-0.129$) and vanishes as $x\to-\infty$.

**Initialization and normalization.** For unit-variance inputs, $\mathbb{E}[\mathrm{GELU}(z)^2]=0.425$ against ReLU's $0.5$, so He-style scaling ($\mathrm{Var}(w)=2/n_{in}$) is a close fit. Transformers rarely depend on that: LayerNorm before each block re-standardises the inputs, which keeps them in the region $|x|\lesssim3$ where the curve differs from ReLU.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-2., -1., 0., 1., 2.], requires_grad=True)
print(nn.GELU()(x))                      # [-0.0455, -0.1587, 0.0000, 0.8413, 1.9545]
print(F.gelu(x, approximate="tanh"))     # [-0.0454, -0.1588, 0.0000, 0.8412, 1.9546]

def gelu(x):                             # x * Phi(x), with Phi written through erf
    return 0.5 * x * (1 + torch.erf(x / 2**0.5))

gelu(x).sum().backward()
print(x.grad)                            # [-0.0852, -0.0833, 0.5000, 1.0833, 1.0852]  = Phi(x) + x*phi(x)

mlp = nn.Sequential(nn.Linear(768, 3072), nn.GELU(), nn.Linear(3072, 768))  # BERT-base FFN shape
```

Module, functional, tanh variant and a from-scratch version checked by autograd

> **Approximation mismatch:** A checkpoint trained with the tanh form and served with the exact form (or the reverse) changes every activation by up to $4.7\times10^{-4}$. One layer barely notices, but the differences compound over dozens of layers and can shift logits enough to break exact-match tests. Check the config (`hidden_act: gelu` vs `gelu_new`/`gelu_pytorch_tanh`).

> **GELU and Swish are near twins:** $x\,\sigma(1.702x)$ is a Swish with $\beta=1.702$. Its minimum is $-0.164$ at $x=-0.751$, against GELU's $-0.170$ at $-0.752$. Any difference in results between GELU and Swish/SiLU comes from this small shape difference, not from a different mechanism.

**Strengths**

- Smooth everywhere, so gradients and second-order information are well defined.
- Small gradients still reach negative pre-activations: no dead units.
- The established default for transformers; matching it keeps you compatible with pretrained checkpoints.

**Limitations**

- Costs more than ReLU (erf or tanh per element), which matters on small edge devices without fused kernels.
- Non-monotonic: a slight negative slope for x < −0.752.
- Two incompatible variants (exact and tanh) that are easy to mix up.
- Gains over ReLU outside transformers are modest and setup-dependent.

**Question:** A GELU unit receives x = −0.5 and an upstream gradient of 2. What gradient does it pass back to x?

- A. 0, as with ReLU
- B. ≈ 0.265
- C. ≈ −0.31
- D. 1.0

**Answer:** B. ≈ 0.265 — $\mathrm{GELU}'(-0.5)=\Phi(-0.5)+(-0.5)\varphi(-0.5)=0.3085-0.1760=0.1325$, so it passes $2\times0.1325=0.265$. ReLU would pass 0. Option $-0.31$ is $2\cdot\mathrm{GELU}(-0.5)$, the output rather than the slope.

**Question:** In the stochastic-gate view, what does the network compute at inference time?

- A. A fresh random Bernoulli(Φ(x)) mask on every forward pass
- B. The expected value x·Φ(x), with no randomness
- C. The input x whenever Φ(x) > 0.5, and 0 otherwise
- D. The input multiplied by a fixed keep probability, as in dropout

**Answer:** B. The expected value x·Φ(x), with no randomness — GELU is the deterministic expectation $\mathbb{E}[x\,m]=x\Phi(x)$ of the gate; no sampling happens in training or inference. Option 3 is ReLU's hard decision, and option 4 is dropout, whose keep probability does not depend on $x$.

**Key points**

- $\mathrm{GELU}(x)=x\Phi(x)$: a smooth ReLU that gates by probability rather than by sign.
- $\mathrm{GELU}'(x)=\Phi(x)+x\varphi(x)$, in $[-0.129,\,1.129]$; $\mathrm{GELU}'(x)+\mathrm{GELU}'(-x)=1$.
- Non-monotonic: minimum $-0.1700$ at $x=-0.7518$.
- It is the expected value of $x\cdot m$ with $m\sim\mathrm{Bernoulli}(\Phi(x))$, which links it to dropout and zoneout.
- Approximations: tanh form (max error $4.7\times10^{-4}$) and $x\sigma(1.702x)$ (max error $0.020$).
- Used in BERT, GPT-2/3 and ViT; match the variant (exact or tanh) that a checkpoint was trained with.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
