# ReLU

> max(0, x): slope exactly 1 where active and 0 elsewhere. Cheap, non-saturating and sparse, it made deep nets trainable, but a unit that is off for every input stops learning.

Source: https://www.mlfoundry.in/learn/relu  
Section: Deep Learning › Activation Functions  
Level: core · 9 min read · updated 2026-10-11

**ReLU:** (rectified linear unit) is the activation $\mathrm{ReLU}(x)=\max(0,x)$. It passes positive pre-activations through unchanged and sets negative ones to zero, so a unit is either **on** (slope 1) or **off** (output and slope 0).

ReLU is the default hidden activation of modern deep learning. It replaced sigmoid and tanh because it does not saturate for positive inputs, costs one comparison, and makes many units exactly zero. It is also the simplest nonlinearity that still gives a network its expressive power.

The function is old. Fukushima used rectifying units in his 1969 visual feature extractor and in the cognitron. The modern revival came in three steps. Nair and Hinton (2010) used rectified linear units in restricted Boltzmann machines. Glorot, Bordes and Bengio (2011) showed that deep rectifier networks train well **without** unsupervised pre-training and produce sparse representations. Krizhevsky, Sutskever and Hinton's AlexNet (2012) reported that a 4-layer CNN with ReLUs reached 25% training error on CIFAR-10 six times faster than the same network with tanh.

## Formula and derivative

$$
\mathrm{ReLU}(x)=\max(0,x)=\begin{cases}x & x>0\\ 0 & x\le 0\end{cases}\qquad\qquad \mathrm{ReLU}'(x)=\begin{cases}1 & x>0\\ 0 & x<0\end{cases}
$$

- $x$: the unit's pre-activation $w^\top a+b$
- $\mathrm{ReLU}'(0)$: undefined; any value in $[0,1]$ is a valid subgradient. PyTorch, TensorFlow and JAX use 0
- $\mathrm{ReLU}'(x)$: the Heaviside step $\mathbb{1}[x>0]$, a gate that either passes the backpropagated error or blocks it

In backprop the derivative acts as a mask: $\delta_j=\big(W^\top\delta^{(l+1)}\big)_j\cdot\mathbb{1}[z_j>0]$. Active units pass their error back **unscaled**. Inactive units pass nothing, and their incoming weights get zero gradient from that example. The kink at 0 is hit with probability zero for real-valued inputs, so the subgradient choice almost never matters.

Close relatives: **ReLU6** $=\min(\max(0,x),6)$, used in MobileNets so activations fit low-precision formats; **softplus** $\log(1+e^x)$, a smooth version whose derivative is the sigmoid; and the leaky family (**Leaky ReLU**, **PReLU**, **ELU**), which give the negative side a nonzero slope.

**Properties**

- **Range:** $[0,\infty)$
- **Monotonic:** yes (non-decreasing)
- **Smooth:** continuous, piecewise linear; not differentiable at 0
- **Zero-centred:** no: for $x\sim\mathcal N(0,1)$ the mean output is $1/\sqrt{2\pi}=0.399$
- **Saturating:** not for $x>0$ (slope 1 forever); fully flat for $x<0$
- **Sparsity:** exactly half the outputs are 0 for a symmetric zero-mean input
- **Cost:** one comparison forward, one mask backward; no exp

> Figure: ReLU and its step derivative, with Leaky ReLU and softplus for comparison. Move the probe: the slope is exactly 1 anywhere on the right and exactly 0 anywhere on the left. The output statistics show the positive mean (0.399 for standard normal inputs) and the 50% of exact zeros.

## Why it trains: no saturation on the active side

Backprop multiplies one $\varphi'(z)$ per layer. Sigmoid's slope is at most 0.25 and tanh's falls toward 0 for large $|z|$, so deep stacks of them shrink the gradient exponentially (see **Vanishing gradients**). An active ReLU contributes a factor of exactly 1, however large $z$ is. The only shrinkage left comes from the weights and from the units that are off, and both are handled by the initialization.

For a zero-mean symmetric pre-activation, ReLU keeps half the second moment: $\mathbb E[\mathrm{ReLU}(z)^2]=\tfrac12\operatorname{Var}(z)$. To keep the signal's scale constant through depth, the weights need twice the variance that a linear layer would: this is **He (Kaiming) initialization**.

$$
\operatorname{Var}(z^{(l+1)})=n_{in}\operatorname{Var}(w)\,\mathbb E\big[\mathrm{ReLU}(z^{(l)})^2\big]=n_{in}\operatorname{Var}(w)\cdot\tfrac12\operatorname{Var}(z^{(l)})\;\Rightarrow\;\operatorname{Var}(w)=\frac{2}{n_{in}}
$$

- $n_{in}$: fan-in of the layer
- $\tfrac12$: the share of $\mathbb E[z^2]$ that survives rectification (half the inputs are zeroed)

## A ReLU network is a sum of hinges

With one hidden layer and a scalar input, each unit contributes a **hinge** $v_j\max(0,w_jx+b_j)$: zero on one side of its kink at $x=-b_j/w_j$ and a straight line on the other. The output is their sum, so it is continuous and piecewise linear with at most one kink per unit. Training moves the kinks and the slopes until the polyline follows the data.

$$
\hat y(x)=c+\sum_{j=1}^{H} v_j\,\max(0,\,w_jx+b_j)
$$

- $-b_j/w_j$: the kink of unit $j$; $w_j>0$ means the unit is active to the right of it
- $v_j$: the hinge's output weight: how much slope the unit adds once active
- $H$: hidden units: at most $H$ kinks, so at most $H+1$ linear pieces

Deeper ReLU networks are still piecewise linear in their input, but composition folds the space, and the number of linear regions can grow exponentially with depth (Montúfar et al., 2014). That is one formal sense in which depth buys expressive power.

> Figure: An 8-unit ReLU network fit to 25 noisy points by full-batch gradient descent (3000 steps). Thin lines are the individual hinges, the violet line is their sum, and ▸/◂ on the axis mark each kink and which side is active. At the default learning rate 0.03 all 8 units stay alive and the MSE reaches 0.0031. Raise the learning rate to 0.10: within the first 20 steps the large early updates push 3 units off every training point, they never come back, and the fit stalls at 0.0173. A bias shift of −3 kills 3 units before training even starts.

## Worked example: three hinges

Take $w=(1,\,-1,\,0.5)$, $b=(-0.5,\,1,\,-1)$, $v=(1.2,\,-0.8,\,2)$ and $c=0.1$. The kinks are at $x=0.5$, $1$ and $2$. On each piece, the slope of $\hat y$ is the sum of $v_jw_j$ over the active units:

Which units are active, and the resulting slope of ŷ (checked numerically in Node)

| x range | active units | slope of ŷ | ŷ at a point |
| --- | --- | --- | --- |
| $x<0.5$ | 2 | $(-0.8)(-1)=0.8$ | $\hat y(0)=-0.7$ |
| $0.5<x<1$ | 1, 2 | $1.2+0.8=2.0$ | $\hat y(0.75)=0.2$ |
| $1<x<2$ | 1 | $1.2$ | $\hat y(1.5)=1.3$ |
| $x>2$ | 1, 3 | $1.2+2\cdot0.5=2.2$ | $\hat y(3)=4.1$ |

**Forward and backward at x = 1.5, target y = 1, L = ½(ŷ − y)²**

1. Pre-activations $z=wx+b=(1.0,\,-0.5,\,-0.25)$, so $h=\mathrm{ReLU}(z)=(1.0,\,0,\,0)$ and $\hat y=0.1+1.2\cdot1.0=1.3$. Loss $0.045$.
2. Output error $\partial L/\partial\hat y=\hat y-y=0.3$, so $\partial L/\partial v=0.3\,h=(0.3,\,0,\,0)$ and $\partial L/\partial c=0.3$.
3. Hidden errors $\delta_j=v_j\cdot0.3\cdot\mathbb 1[z_j>0]=(0.36,\,0,\,0)$: units 2 and 3 are off, so the gate blocks them.
4. Weight gradients $\partial L/\partial w_j=\delta_j x=(0.54,\,0,\,0)$ and $\partial L/\partial b_j=\delta_j=(0.36,\,0,\,0)$. Only unit 1 learns from this example.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-2.0, -0.5, 0.0, 0.5, 2.0], requires_grad=True)

y = nn.ReLU()(x)               # module, for nn.Sequential
y = F.relu(x)                  # functional; F.relu_(t) / nn.ReLU(inplace=True) overwrite t
y.sum().backward()
print(y.tolist())              # [0.0, 0.0, 0.0, 0.5, 2.0]
print(x.grad.tolist())         # [0.0, 0.0, 0.0, 1.0, 1.0]   <- f'(0) is taken as 0

# From scratch: forward is a clamp, backward is a mask
class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        ctx.save_for_backward(x)
        return x.clamp(min=0)
    @staticmethod
    def backward(ctx, g):
        (x,) = ctx.saved_tensors
        return g * (x > 0).to(g.dtype)

# Dead units: zero on every example of a (large) batch
lin = nn.Linear(64, 256)
nn.init.kaiming_normal_(lin.weight, nonlinearity="relu")   # Var(w) = 2 / fan_in
z = lin(torch.randn(4096, 64))
dead = (z <= 0).all(dim=0)
print(f"dead units: {dead.sum().item()} / {z.shape[1]}")
```

ReLU three ways, plus a dead-unit check

> **Dying ReLU:** A unit whose pre-activation is $\le0$ on **every** input outputs 0 and has $\mathrm{ReLU}'=0$ everywhere, so its weights and bias never receive gradient again. Typical causes are a learning rate that is too high (one big step throws the bias far negative, as in the figure), a large negative bias, or an unlucky initialization. Watch the share of always-zero units per layer; fixes are a smaller learning rate or warm-up, He init with zero biases, normalization layers, or a leaky variant.

> **Sparsity is a feature:** For zero-mean inputs about half the units are exactly zero on any given example, and different examples switch on different subsets. Glorot et al. (2011) argued this sparse, input-dependent gating is part of why rectifiers work: each example effectively uses a different linear sub-network, and the zeros are free to compute and to store.

> **Where you meet it:** AlexNet, VGG and ResNet use ReLU throughout, and so did the feed-forward block of the original Transformer, $\max(0,xW_1+b_1)W_2+b_2$. Most recent language models switched to GELU or gated SiLU variants, but ReLU is still the default first choice for CNNs and MLPs.

**Strengths**

- Gradient exactly 1 on the active side: no saturation, healthy gradient flow in deep nets
- Cheapest nonlinearity: a comparison forward, a mask backward
- Sparse activations: about half the units are zero for a given input
- Piecewise linear: easy to analyse, and compatible with He init

**Limitations**

- Not zero-centred: mean output $0.399\sigma$ shifts the next layer's inputs
- Dying units: zero gradient for all inputs once a unit is off everywhere
- Not differentiable at 0, and the kinks make the loss surface non-smooth
- Unbounded output: activations can grow without normalization

**Question:** A ReLU unit has z ≤ 0 for every example in the training set. What happens to its incoming weights under gradient descent?

- A. They never change: every gradient through the unit is zero
- B. They shrink toward zero
- C. They move slowly because the slope is small but nonzero
- D. They change only when the learning rate is large

**Answer:** A. They never change: every gradient through the unit is zero — $\partial L/\partial w_j=\delta_j\,a$ with $\delta_j=(W^\top\delta)_j\cdot\mathrm{ReLU}'(z_j)$, and $\mathrm{ReLU}'(z_j)=0$ on every example. The unit is dead: no example can move it back. Weight decay is the only thing still acting on it, and that only shrinks the weights toward zero.

**Question:** A 1-input network has one hidden layer of 8 ReLU units and a linear output. At most how many linear pieces can its output have?

- A. 8
- B. 9
- C. 16
- D. 256

**Answer:** B. 9 — Each unit adds one hinge with one kink at $-b_j/w_j$. Eight kinks split the real line into at most 9 intervals, and the output is linear on each. Depth, not width, is what multiplies the number of regions.

**Key points**

- $\mathrm{ReLU}(x)=\max(0,x)$, $\mathrm{ReLU}'(x)=\mathbb 1[x>0]$ (frameworks use 0 at the kink).
- Active units pass gradient with factor exactly 1, so ReLU avoids the saturation that made deep sigmoid/tanh nets hard to train.
- Pair it with He init, $\operatorname{Var}(w)=2/n_{in}$, because rectification keeps half the second moment.
- A one-hidden-layer ReLU net is a sum of hinges: continuous and piecewise linear, at most $H+1$ pieces in 1-D.
- Dying ReLU: a unit off on every input never gets gradient again; too-high learning rates and negative biases cause it.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
