Definition
GELU (Gaussian Error Linear Unit) is the activation , where is the standard normal CDF. It scales each input by how likely a standard normal variable is to be smaller than it, instead of gating it by its sign as ReLU does.
ReLU multiplies its input by a hard 0/1 gate, . GELU swaps the hard gate for a soft one, . Large positive inputs pass almost unchanged and large negative inputs are almost zeroed, as with ReLU. Near zero the gate is partial, which makes the curve smooth and lets it dip slightly below zero.
Hendrycks & Gimpel proposed it in 2016 ("Gaussian Error Linear Units"). It became the standard feed-forward activation of transformers: BERT, GPT-2, GPT-3 and the Vision Transformer (ViT) all use it.
Formula and derivative
- standard normal CDF, for
- the error function,
The product rule gives the derivative. The second term is what lets the slope go negative and rise slightly above 1:
- standard normal density, (note and )
- the gate itself: the slope ReLU would have if its gate were frozen
- the extra term from the gate changing with x; negative for x < 0
Setting and solving numerically gives the dip: the minimum is at . The slope peaks at at and is lowest, , at , where . Since (because ), the slopes at always sum to 1.
At a glance
- Range
- ; unbounded above, bounded below
- Monotonic
- No. Decreasing on , increasing after
- Smooth
- Yes, infinitely differentiable; ,
- Zero-centred
- No. For the output has mean and std
- Saturation
- Only on the negative side ( as ); no saturation for positive inputs
- Cost
- One erf (or one tanh in the approximation) per element: more than ReLU's compare, though usually fused into the matmul epilogue
Where it comes from: an expected stochastic gate
Hendrycks & Gimpel derived GELU from a random regulariser. Multiply the input by a mask , so an input is kept with probability . Small inputs are usually dropped, large ones usually kept. The deterministic activation is the expected output:
- random keep-mask, with probability
- keep probability: since pre-activations are often roughly standard normal (e.g. after normalization), it is the chance that a typical input is smaller than this one
This combines two existing ideas. Dropout multiplies by a random mask whose keep probability is fixed. Zoneout (for RNNs) randomly keeps the previous hidden state. ReLU applies a deterministic mask, . GELU's mask is random but depends on the input, and the network uses its average.
The two approximations
An erf was not always a fast primitive, so the paper also gives two cheaper forms:
- tanh approximation; max absolute error (at )
- sigmoid approximation, i.e. Swish with ; max absolute error (at )
The original BERT and GPT-2 code used the tanh form, which is why libraries keep it: PyTorch's approximate="tanh" and Hugging Face's gelu_new exist to reproduce those checkpoints exactly. On GPUs the elementwise op is memory-bound, so for a new model the exact erf form typically costs about the same.
Worked example
| x | Φ(x) | GELU(x) | tanh approx | x·σ(1.702x) | GELU′(x) |
|---|---|---|---|---|---|
| −3 | 0.0013 | −0.0040 | −0.0036 | −0.0181 | −0.0119 |
| −2 | 0.0228 | −0.0455 | −0.0454 | −0.0643 | −0.0852 |
| −1 | 0.1587 | −0.1587 | −0.1588 | −0.1542 | −0.0833 |
| −0.7518 | 0.2261 | −0.1700 | −0.1700 | −0.1636 | 0.0000 |
| −0.5 | 0.3085 | −0.1543 | −0.1543 | −0.1496 | 0.1325 |
| 0 | 0.5000 | 0.0000 | 0.0000 | 0.0000 | 0.5000 |
| 0.5 | 0.6915 | 0.3457 | 0.3457 | 0.3504 | 0.8675 |
| 1 | 0.8413 | 0.8413 | 0.8412 | 0.8458 | 1.0833 |
| 2 | 0.9772 | 1.9545 | 1.9546 | 1.9357 | 1.0852 |
Two readings. At , GELU equals , which is just . And for any between and the slope is positive but well below 1. At a ReLU unit sends back no gradient, while a GELU unit sends back times the upstream gradient.
Effect on training
Gradient flow. The slope is never exactly zero on an interval, so GELU has no "dead" units in ReLU's sense. Negative pre-activations still receive small gradients. For the slope is negative, so the gradient reaching the input has the opposite sign to the upstream one, but it is tiny (the slope is never below ) and vanishes as .
Initialization and normalization. For unit-variance inputs, against ReLU's , so He-style scaling () is a close fit. Transformers rarely depend on that: LayerNorm before each block re-standardises the inputs, which keeps them in the region where the curve differs from ReLU.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([-2., -1., 0., 1., 2.], requires_grad=True)4print(nn.GELU()(x)) # [-0.0455, -0.1587, 0.0000, 0.8413, 1.9545]5print(F.gelu(x, approximate="tanh")) # [-0.0454, -0.1588, 0.0000, 0.8412, 1.9546]67def gelu(x): # x * Phi(x), with Phi written through erf8 return 0.5 * x * (1 + torch.erf(x / 2**0.5))910gelu(x).sum().backward()11print(x.grad) # [-0.0852, -0.0833, 0.5000, 1.0833, 1.0852] = Phi(x) + x*phi(x)1213mlp = nn.Sequential(nn.Linear(768, 3072), nn.GELU(), nn.Linear(3072, 768)) # BERT-base FFN shapeUse when
- +Smooth everywhere, so gradients and second-order information are well defined.
- +Small gradients still reach negative pre-activations: no dead units.
- +The established default for transformers; matching it keeps you compatible with pretrained checkpoints.
Avoid when
- −Costs more than ReLU (erf or tanh per element), which matters on small edge devices without fused kernels.
- −Non-monotonic: a slight negative slope for x < −0.752.
- −Two incompatible variants (exact and tanh) that are easy to mix up.
- −Gains over ReLU outside transformers are modest and setup-dependent.
Check yourself
A GELU unit receives x = −0.5 and an upstream gradient of 2. What gradient does it pass back to x?
Check yourself
In the stochastic-gate view, what does the network compute at inference time?
Remember
- ✓: a smooth ReLU that gates by probability rather than by sign.
- ✓, in ; .
- ✓Non-monotonic: minimum at .
- ✓It is the expected value of with , which links it to dropout and zoneout.
- ✓Approximations: tanh form (max error ) and (max error ).
- ✓Used in BERT, GPT-2/3 and ViT; match the variant (exact or tanh) that a checkpoint was trained with.