Depth only helps if something bends between the layers. Compose two linear layers and you get one linear layer back:
activation
compare
range [0, ∞)monotonic yessmooth nozero-centred no
Kink at 0; f′(0) is taken as 0 (the PyTorch convention).
E[f(x)] 0.399Var[f(x)] 0.341E[f′(x)] 0.500f′ = 0 for 50.0% of inputsE[f′]¹⁰ 9.8e−4
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
The common choices
ReLU — cheap, gradient exactly 1 for . The default for MLPs and CNNs.
Sigmoid — output in , maximum gradient . Use it for binary outputs, not hidden layers.
Tanh — a zero-centred sigmoid with range ; it still saturates for large .
GELU — a smooth ReLU; standard in transformers such as BERT and GPT.
All 18 at a glance
| Activation | Formula | Range | Typical use |
|---|---|---|---|
| Binary step | Historical perceptron; no useful gradient | ||
| Linear | Regression outputs; never between hidden layers | ||
| Sigmoid | Binary outputs, gates in LSTMs/GRUs | ||
| Tanh | RNN hidden states; zero-centred | ||
| ReLU | Default for MLPs and CNNs | ||
| Leaky ReLU | , | Avoiding dead units; GAN discriminators | |
| PReLU | , learned | CNNs where the negative slope is learned | |
| ELU | if , else | Smoother negative side, mean closer to 0 | |
| SELU | Self-normalizing MLPs (with LeCun init) | ||
| GELU | Transformers (BERT, GPT) | ||
| Swish / SiLU | EfficientNet, many modern CNNs | ||
| Mish | YOLOv4-style detectors | ||
| Softplus | Positive outputs (variances, rates) | ||
| Softsign | Gentler-saturating tanh alternative | ||
| Softmax | , sums to 1 | Multi-class outputs, attention weights | |
| GLU | Gated conv / language models | ||
| SwiGLU | Transformer FFNs (LLaMA, PaLM) | ||
| GeGLU | Transformer FFNs (T5 v1.1) |
Each of these has its own article in the Activation Functions module, with its derivative, failure modes and a worked example.
python
1import torch.nn as nn23mlp = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))Check yourself
What is the maximum value of the sigmoid's derivative?
Remember
- ✓Linear ∘ linear = linear.
- ✓ReLU for hidden layers, GELU in transformers.
- ✓Saturating activations cause vanishing gradients.