GELU

Weight each input by the probability that a standard normal falls below it: x·Φ(x), a smooth ReLU and the default activation of transformers.

8 min read640 wordscore0 viewsupdated 2026-10-11

Definition

GELU (Gaussian Error Linear Unit) is the activation GELU(x)=x Φ(x)\mathrm{GELU}(x)=x\,\Phi(x), where Φ\Phi is the standard normal CDF. It scales each input by how likely a standard normal variable is to be smaller than it, instead of gating it by its sign as ReLU does.

ReLU multiplies its input by a hard 0/1 gate, 1[x>0]\mathbb{1}[x>0]. GELU swaps the hard gate for a soft one, Φ(x)∈(0,1)\Phi(x)\in(0,1). Large positive inputs pass almost unchanged and large negative inputs are almost zeroed, as with ReLU. Near zero the gate is partial, which makes the curve smooth and lets it dip slightly below zero.

Hendrycks & Gimpel proposed it in 2016 ("Gaussian Error Linear Units"). It became the standard feed-forward activation of transformers: BERT, GPT-2, GPT-3 and the Vision Transformer (ViT) all use it.

Formula and derivative

GELU(x)=x Φ(x)=x2(1+erf⁡(x/2))\mathrm{GELU}(x)=x\,\Phi(x)=\frac{x}{2}\Big(1+\operatorname{erf}\big(x/\sqrt2\big)\Big)
Φ(x)\Phi(x)
standard normal CDF, P(Z≤x)P(Z\le x) for Z∼N(0,1)Z\sim\mathcal N(0,1)
erf⁡\operatorname{erf}
the error function, Φ(x)=12(1+erf⁡(x/2))\Phi(x)=\tfrac12\big(1+\operatorname{erf}(x/\sqrt2)\big)

The product rule gives the derivative. The second term is what lets the slope go negative and rise slightly above 1:

GELU′(x)=Φ(x)+x φ(x),GELU′′(x)=φ(x) (2−x2)\mathrm{GELU}'(x)=\Phi(x)+x\,\varphi(x),\qquad \mathrm{GELU}''(x)=\varphi(x)\,(2-x^2)
φ(x)\varphi(x)
standard normal density, e−x2/2/2πe^{-x^2/2}/\sqrt{2\pi} (note Φ′=φ\Phi'=\varphi and φ′(x)=−xφ(x)\varphi'(x)=-x\varphi(x))
Φ(x)\Phi(x)
the gate itself: the slope ReLU would have if its gate were frozen
xφ(x)x\varphi(x)
the extra term from the gate changing with x; negative for x < 0

Setting GELU′(x)=0\mathrm{GELU}'(x)=0 and solving numerically gives the dip: the minimum is −0.1700-0.1700 at x=−0.7518x=-0.7518. The slope peaks at 1.12891.1289 at x=2x=\sqrt2 and is lowest, −0.1289-0.1289, at x=−2x=-\sqrt2, where GELU′′=0\mathrm{GELU}''=0. Since GELU(x)−GELU(−x)=x\mathrm{GELU}(x)-\mathrm{GELU}(-x)=x (because Φ(−x)=1−Φ(x)\Phi(-x)=1-\Phi(x)), the slopes at ±x\pm x always sum to 1.

At a glance

Range
[−0.170, ∞)[-0.170,\ \infty); unbounded above, bounded below
Monotonic
No. Decreasing on (−∞,−0.752)(-\infty,-0.752), increasing after
Smooth
Yes, infinitely differentiable; GELU′(0)=0.5\mathrm{GELU}'(0)=0.5, GELU′′(0)=2/π≈0.798\mathrm{GELU}''(0)=\sqrt{2/\pi}\approx0.798
Zero-centred
No. For x∼N(0,1)x\sim\mathcal N(0,1) the output has mean 1/(2π)≈0.2821/(2\sqrt\pi)\approx0.282 and std 0.5880.588
Saturation
Only on the negative side (→0\to0 as x→−∞x\to-\infty); no saturation for positive inputs
Cost
One erf (or one tanh in the approximation) per element: more than ReLU's compare, though usually fused into the matmul epilogue

Where it comes from: an expected stochastic gate

Hendrycks & Gimpel derived GELU from a random regulariser. Multiply the input by a mask m∼Bernoulli(Φ(x))m\sim\mathrm{Bernoulli}(\Phi(x)), so an input is kept with probability Φ(x)\Phi(x). Small inputs are usually dropped, large ones usually kept. The deterministic activation is the expected output:

E[ x⋅m ]=x⋅P(m=1)+0⋅P(m=0)=x Φ(x)\mathbb{E}[\,x\cdot m\,]=x\cdot P(m=1)+0\cdot P(m=0)=x\,\Phi(x)
mm
random keep-mask, m=1m=1 with probability Φ(x)\Phi(x)
Φ(x)\Phi(x)
keep probability: since pre-activations are often roughly standard normal (e.g. after normalization), it is the chance that a typical input is smaller than this one

This combines two existing ideas. Dropout multiplies by a random mask whose keep probability is fixed. Zoneout (for RNNs) randomly keeps the previous hidden state. ReLU applies a deterministic mask, 1[x>0]\mathbb{1}[x>0]. GELU's mask is random but depends on the input, and the network uses its average.

x·m with m ~ Bernoulli(Φ(x)) · 1 draw per probe-10123-3-2-10123m=1: xm=0: 0zoom: negative side-0.3-0.2-0.100.10.2-3-2-10min −0.170 at −0.752tanh approx error: max 4.7e-4 at ±2.70−4.7e-44.7e-4-6-3036x·σ(1.702x) error: max 0.020 at ±2.27−0.0200.020-6-3036● kept (x) ○ dropped (0) in the latest draw ■ running mean — GELU x·Φ(x)
draws n 1max |mean − GELU| 2.742largest std. error |x|·√(Φ(1−Φ)/n) 0.384at x = −0.75: mean 0.000 vs -0.170 (keep prob 0.227)
fig.Each draw samples one Bernoulli(Φ(x)) mask per probe: the output is either x (orange dot on the dashed identity line) or 0 (hollow dot). Blue squares are running means over all draws. Press play: the means settle onto x·Φ(x) at the 1/√n rate, and the zoom on the right shows them finding the dip near −0.75. The bottom panels plot the error of the two closed-form approximations.
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 0.841 · f′(x) = 1.083−1012345−4−2024x →GELU (exact)GELU f′ReLUSwish
activation
compare
form
range [≈ −0.170, ∞)monotonic nosmooth yeszero-centred no
Exact form by default. approx = 1 switches to ½x(1 + tanh(√(2/π)(x + 0.044715x³))), max error ≈ 5e-4. Minimum ≈ −0.170 at x ≈ −0.752.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no sampling↑ 40%E[f]−0.171.914.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.282Var[f(x)] 0.346E[f′(x)] 0.500f′ = 0 for 0.0% of inputsE[f′]¹⁰ 9.8e−4
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.GELU against ReLU and Swish. Move the probe into the negative region: ReLU's slope is exactly 0 there, while GELU still passes a small (sometimes negative) gradient.

The two approximations

An erf was not always a fast primitive, so the paper also gives two cheaper forms:

GELU(x)≈12x(1+tanh⁡ ⁣[2/π (x+0.044715 x3)])≈x σ(1.702 x)\mathrm{GELU}(x)\approx\tfrac12x\Big(1+\tanh\!\Big[\sqrt{2/\pi}\,\big(x+0.044715\,x^3\big)\Big]\Big)\approx x\,\sigma(1.702\,x)
tanh⁡[⋅]\tanh[\cdot]
tanh approximation; max absolute error 4.7×10−44.7\times10^{-4} (at x≈±2.70x\approx\pm2.70)
σ(1.702x)\sigma(1.702x)
sigmoid approximation, i.e. Swish with β=1.702\beta=1.702; max absolute error 0.0200.020 (at x≈±2.27x\approx\pm2.27)

The original BERT and GPT-2 code used the tanh form, which is why libraries keep it: PyTorch's approximate="tanh" and Hugging Face's gelu_new exist to reproduce those checkpoints exactly. On GPUs the elementwise op is memory-bound, so for a new model the exact erf form typically costs about the same.

Worked example

xΦ(x)GELU(x)tanh approxx·σ(1.702x)GELU′(x)
−30.0013−0.0040−0.0036−0.0181−0.0119
−20.0228−0.0455−0.0454−0.0643−0.0852
−10.1587−0.1587−0.1588−0.1542−0.0833
−0.75180.2261−0.1700−0.1700−0.16360.0000
−0.50.3085−0.1543−0.1543−0.14960.1325
00.50000.00000.00000.00000.5000
0.50.69150.34570.34570.35040.8675
10.84130.84130.84120.84581.0833
20.97721.95451.95461.93571.0852
GELU, its approximations and its slope at a few inputs (4 decimals; the highlighted row is the minimum)

Two readings. At x=−1x=-1, GELU equals −Φ(−1)=−0.1587-\Phi(-1)=-0.1587, which is just xΦ(x)x\Phi(x). And for any xx between −0.75-0.75 and 00 the slope is positive but well below 1. At x=−0.5x=-0.5 a ReLU unit sends back no gradient, while a GELU unit sends back 0.13250.1325 times the upstream gradient.

Effect on training

Gradient flow. The slope is never exactly zero on an interval, so GELU has no "dead" units in ReLU's sense. Negative pre-activations still receive small gradients. For x<−0.752x<-0.752 the slope is negative, so the gradient reaching the input has the opposite sign to the upstream one, but it is tiny (the slope is never below −0.129-0.129) and vanishes as x→−∞x\to-\infty.

Initialization and normalization. For unit-variance inputs, E[GELU(z)2]=0.425\mathbb{E}[\mathrm{GELU}(z)^2]=0.425 against ReLU's 0.50.5, so He-style scaling (Var(w)=2/nin\mathrm{Var}(w)=2/n_{in}) is a close fit. Transformers rarely depend on that: LayerNorm before each block re-standardises the inputs, which keeps them in the region ∣x∣≲3|x|\lesssim3 where the curve differs from ReLU.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-2., -1., 0., 1., 2.], requires_grad=True)
4print(nn.GELU()(x)) # [-0.0455, -0.1587, 0.0000, 0.8413, 1.9545]
5print(F.gelu(x, approximate="tanh")) # [-0.0454, -0.1588, 0.0000, 0.8412, 1.9546]
6
7def gelu(x): # x * Phi(x), with Phi written through erf
8 return 0.5 * x * (1 + torch.erf(x / 2**0.5))
9
10gelu(x).sum().backward()
11print(x.grad) # [-0.0852, -0.0833, 0.5000, 1.0833, 1.0852] = Phi(x) + x*phi(x)
12
13mlp = nn.Sequential(nn.Linear(768, 3072), nn.GELU(), nn.Linear(3072, 768)) # BERT-base FFN shape
Module, functional, tanh variant and a from-scratch version checked by autograd

Use when

  • +Smooth everywhere, so gradients and second-order information are well defined.
  • +Small gradients still reach negative pre-activations: no dead units.
  • +The established default for transformers; matching it keeps you compatible with pretrained checkpoints.

Avoid when

  • −Costs more than ReLU (erf or tanh per element), which matters on small edge devices without fused kernels.
  • −Non-monotonic: a slight negative slope for x < −0.752.
  • −Two incompatible variants (exact and tanh) that are easy to mix up.
  • −Gains over ReLU outside transformers are modest and setup-dependent.

Check yourself

A GELU unit receives x = −0.5 and an upstream gradient of 2. What gradient does it pass back to x?

Check yourself

In the stochastic-gate view, what does the network compute at inference time?

Remember

  • ✓GELU(x)=xΦ(x)\mathrm{GELU}(x)=x\Phi(x): a smooth ReLU that gates by probability rather than by sign.
  • ✓GELU′(x)=Φ(x)+xφ(x)\mathrm{GELU}'(x)=\Phi(x)+x\varphi(x), in [−0.129, 1.129][-0.129,\,1.129]; GELU′(x)+GELU′(−x)=1\mathrm{GELU}'(x)+\mathrm{GELU}'(-x)=1.
  • ✓Non-monotonic: minimum −0.1700-0.1700 at x=−0.7518x=-0.7518.
  • ✓It is the expected value of x⋅mx\cdot m with m∼Bernoulli(Φ(x))m\sim\mathrm{Bernoulli}(\Phi(x)), which links it to dropout and zoneout.
  • ✓Approximations: tanh form (max error 4.7×10−44.7\times10^{-4}) and xσ(1.702x)x\sigma(1.702x) (max error 0.0200.020).
  • ✓Used in BERT, GPT-2/3 and ViT; match the variant (exact or tanh) that a checkpoint was trained with.
← back to Deep Learning roadmap