# Activation Functions

> Without a nonlinearity, any stack of linear layers collapses into a single linear layer.

Source: https://www.mlfoundry.in/learn/activation-functions  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 6 min read · updated 2026-10-11

Depth only helps if something bends between the layers. Compose two linear layers and you get one linear layer back:

$$
W_2(W_1x+b_1)+b_2=(W_2W_1)\,x+(W_2b_1+b_2)
$$

> Figure: The four activations you will meet most often, drawn on the same axes.

> Figure: Pick any activation, tune its parameters and read f(x) and f′(x) at the probe. The lower panel pushes Gaussian inputs through it: watch E[f′] and the share of zero-gradient inputs — those set how gradients survive depth.

## The common choices

**ReLU** $\max(0,x)$ — cheap, gradient exactly 1 for $x>0$. The default for MLPs and CNNs.

**Sigmoid** $\sigma(x)=1/(1+e^{-x})$ — output in $(0,1)$, maximum gradient $0.25$. Use it for binary outputs, not hidden layers.

**Tanh** — a zero-centred sigmoid with range $(-1,1)$; it still saturates for large $|x|$.

**GELU** $x\,\Phi(x)$ — a smooth ReLU; standard in transformers such as BERT and GPT.

## All 18 at a glance

Activation functions covered in this course. Φ is the standard normal CDF, σ the sigmoid, ⊙ elementwise product.

| Activation | Formula | Range | Typical use |
| --- | --- | --- | --- |
| Binary step | $\mathbb{1}[x\ge 0]$ | $\{0,1\}$ | Historical perceptron; no useful gradient |
| Linear | $ax$ | $(-\infty,\infty)$ | Regression outputs; never between hidden layers |
| Sigmoid | $1/(1+e^{-x})$ | $(0,1)$ | Binary outputs, gates in LSTMs/GRUs |
| Tanh | $\tanh x$ | $(-1,1)$ | RNN hidden states; zero-centred |
| ReLU | $\max(0,x)$ | $[0,\infty)$ | Default for MLPs and CNNs |
| Leaky ReLU | $\max(\alpha x,x)$, $\alpha=0.01$ | $(-\infty,\infty)$ | Avoiding dead units; GAN discriminators |
| PReLU | $\max(ax,x)$, $a$ learned | $(-\infty,\infty)$ | CNNs where the negative slope is learned |
| ELU | $x$ if $x>0$, else $\alpha(e^x-1)$ | $(-\alpha,\infty)$ | Smoother negative side, mean closer to 0 |
| SELU | $\lambda\,\mathrm{ELU}_\alpha(x)$ | $(-\lambda\alpha,\infty)$ | Self-normalizing MLPs (with LeCun init) |
| GELU | $x\,\Phi(x)$ | $[\approx-0.17,\infty)$ | Transformers (BERT, GPT) |
| Swish / SiLU | $x\,\sigma(\beta x)$ | $[\approx-0.28,\infty)$ | EfficientNet, many modern CNNs |
| Mish | $x\tanh(\ln(1+e^x))$ | $[\approx-0.31,\infty)$ | YOLOv4-style detectors |
| Softplus | $\ln(1+e^x)$ | $(0,\infty)$ | Positive outputs (variances, rates) |
| Softsign | $x/(1+\|x\|)$ | $(-1,1)$ | Gentler-saturating tanh alternative |
| Softmax | $e^{x_i}/\sum_j e^{x_j}$ | $(0,1)$, sums to 1 | Multi-class outputs, attention weights |
| GLU | $(xW+b)\odot\sigma(xV+c)$ | $(-\infty,\infty)$ | Gated conv / language models |
| SwiGLU | $\mathrm{Swish}(xW)\odot xV$ | $(-\infty,\infty)$ | Transformer FFNs (LLaMA, PaLM) |
| GeGLU | $\mathrm{GELU}(xW)\odot xV$ | $(-\infty,\infty)$ | Transformer FFNs (T5 v1.1) |

Each of these has its own article in the **Activation Functions** module, with its derivative, failure modes and a worked example.

> **Dying ReLU:** If a unit's input is negative for every example, its gradient is zero and it never recovers. Usual causes: a learning rate that is too large or poor initialization. Fixes: Leaky ReLU, a lower learning rate, He initialization.

```python
import torch.nn as nn

mlp = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
```

**Question:** What is the maximum value of the sigmoid's derivative?

- A. 1
- B. 0.5
- C. 0.25
- D. 0.1

**Answer:** C. 0.25 — $\sigma'(x)=\sigma(x)(1-\sigma(x))$, maximized at $x=0$: $0.5\cdot0.5 = 0.25$. Ten sigmoid layers can shrink a gradient by $0.25^{10}\approx 10^{-6}$.

**Key points**

- Linear ∘ linear = linear.
- ReLU for hidden layers, GELU in transformers.
- Saturating activations cause vanishing gradients.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
