Activation Functions

Without a nonlinearity, any stack of linear layers collapses into a single linear layer.

6 min read156 wordscore2 viewsupdated 2026-10-11

Depth only helps if something bends between the layers. Compose two linear layers and you get one linear layer back:

W2(W1x+b1)+b2=(W2W1) x+(W2b1+b2)W_2(W_1x+b_1)+b_2=(W_2W_1)\,x+(W_2b_1+b_2)
-112ReLUmax(0, x)
fig.The four activations you will meet most often, drawn on the same axes.
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 1.000 · f′(x) = 1.000−1012345−4−2024x →ReLUReLU f′SigmoidTanhGELU
activation
compare
range [0, ∞)monotonic yessmooth nozero-centred no
Kink at 0; f′(0) is taken as 0 (the PyTorch convention).
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no sampling↑ 54%E[f]0.002.004.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.399Var[f(x)] 0.341E[f′(x)] 0.500f′ = 0 for 50.0% of inputsE[f′]¹⁰ 9.8e−4
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.Pick any activation, tune its parameters and read f(x) and f′(x) at the probe. The lower panel pushes Gaussian inputs through it: watch E[f′] and the share of zero-gradient inputs — those set how gradients survive depth.

The common choices

ReLU max⁡(0,x)\max(0,x) — cheap, gradient exactly 1 for x>0x>0. The default for MLPs and CNNs.

Sigmoid σ(x)=1/(1+e−x)\sigma(x)=1/(1+e^{-x}) — output in (0,1)(0,1), maximum gradient 0.250.25. Use it for binary outputs, not hidden layers.

Tanh — a zero-centred sigmoid with range (−1,1)(-1,1); it still saturates for large ∣x∣|x|.

GELU x Φ(x)x\,\Phi(x) — a smooth ReLU; standard in transformers such as BERT and GPT.

All 18 at a glance

ActivationFormulaRangeTypical use
Binary step1[x≥0]\mathbb{1}[x\ge 0]{0,1}\{0,1\}Historical perceptron; no useful gradient
Linearaxax(−∞,∞)(-\infty,\infty)Regression outputs; never between hidden layers
Sigmoid1/(1+e−x)1/(1+e^{-x})(0,1)(0,1)Binary outputs, gates in LSTMs/GRUs
Tanhtanh⁡x\tanh x(−1,1)(-1,1)RNN hidden states; zero-centred
ReLUmax⁡(0,x)\max(0,x)[0,∞)[0,\infty)Default for MLPs and CNNs
Leaky ReLUmax⁡(αx,x)\max(\alpha x,x), α=0.01\alpha=0.01(−∞,∞)(-\infty,\infty)Avoiding dead units; GAN discriminators
PReLUmax⁡(ax,x)\max(ax,x), aa learned(−∞,∞)(-\infty,\infty)CNNs where the negative slope is learned
ELUxx if x>0x>0, else α(ex−1)\alpha(e^x-1)(−α,∞)(-\alpha,\infty)Smoother negative side, mean closer to 0
SELUλ ELUα(x)\lambda\,\mathrm{ELU}_\alpha(x)(−λα,∞)(-\lambda\alpha,\infty)Self-normalizing MLPs (with LeCun init)
GELUx Φ(x)x\,\Phi(x)[≈−0.17,∞)[\approx-0.17,\infty)Transformers (BERT, GPT)
Swish / SiLUx σ(βx)x\,\sigma(\beta x)[≈−0.28,∞)[\approx-0.28,\infty)EfficientNet, many modern CNNs
Mishxtanh⁡(ln⁡(1+ex))x\tanh(\ln(1+e^x))[≈−0.31,∞)[\approx-0.31,\infty)YOLOv4-style detectors
Softplusln⁡(1+ex)\ln(1+e^x)(0,∞)(0,\infty)Positive outputs (variances, rates)
Softsignx/(1+∣x∣)x/(1+|x|)(−1,1)(-1,1)Gentler-saturating tanh alternative
Softmaxexi/∑jexje^{x_i}/\sum_j e^{x_j}(0,1)(0,1), sums to 1Multi-class outputs, attention weights
GLU(xW+b)⊙σ(xV+c)(xW+b)\odot\sigma(xV+c)(−∞,∞)(-\infty,\infty)Gated conv / language models
SwiGLUSwish(xW)⊙xV\mathrm{Swish}(xW)\odot xV(−∞,∞)(-\infty,\infty)Transformer FFNs (LLaMA, PaLM)
GeGLUGELU(xW)⊙xV\mathrm{GELU}(xW)\odot xV(−∞,∞)(-\infty,\infty)Transformer FFNs (T5 v1.1)
Activation functions covered in this course. Φ is the standard normal CDF, σ the sigmoid, ⊙ elementwise product.

Each of these has its own article in the Activation Functions module, with its derivative, failure modes and a worked example.

python
1import torch.nn as nn
2
3mlp = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))

Check yourself

What is the maximum value of the sigmoid's derivative?

Remember

  • ✓Linear ∘ linear = linear.
  • ✓ReLU for hidden layers, GELU in transformers.
  • ✓Saturating activations cause vanishing gradients.
← back to Deep Learning roadmap