ReLU

max(0, x): slope exactly 1 where active and 0 elsewhere. Cheap, non-saturating and sparse, it made deep nets trainable, but a unit that is off for every input stops learning.

9 min read801 wordscore0 viewsupdated 2026-10-11

Definition

ReLU (rectified linear unit) is the activation ReLU(x)=max⁡(0,x)\mathrm{ReLU}(x)=\max(0,x). It passes positive pre-activations through unchanged and sets negative ones to zero, so a unit is either on (slope 1) or off (output and slope 0).

ReLU is the default hidden activation of modern deep learning. It replaced sigmoid and tanh because it does not saturate for positive inputs, costs one comparison, and makes many units exactly zero. It is also the simplest nonlinearity that still gives a network its expressive power.

The function is old. Fukushima used rectifying units in his 1969 visual feature extractor and in the cognitron. The modern revival came in three steps. Nair and Hinton (2010) used rectified linear units in restricted Boltzmann machines. Glorot, Bordes and Bengio (2011) showed that deep rectifier networks train well without unsupervised pre-training and produce sparse representations. Krizhevsky, Sutskever and Hinton's AlexNet (2012) reported that a 4-layer CNN with ReLUs reached 25% training error on CIFAR-10 six times faster than the same network with tanh.

Formula and derivative

ReLU(x)=max⁡(0,x)={xx>00x≤0ReLU′(x)={1x>00x<0\mathrm{ReLU}(x)=\max(0,x)=\begin{cases}x & x>0\\ 0 & x\le 0\end{cases}\qquad\qquad \mathrm{ReLU}'(x)=\begin{cases}1 & x>0\\ 0 & x<0\end{cases}
xx
the unit's pre-activation w⊤a+bw^\top a+b
ReLU′(0)\mathrm{ReLU}'(0)
undefined; any value in [0,1][0,1] is a valid subgradient. PyTorch, TensorFlow and JAX use 0
ReLU′(x)\mathrm{ReLU}'(x)
the Heaviside step 1[x>0]\mathbb{1}[x>0], a gate that either passes the backpropagated error or blocks it

In backprop the derivative acts as a mask: δj=(W⊤δ(l+1))j⋅1[zj>0]\delta_j=\big(W^\top\delta^{(l+1)}\big)_j\cdot\mathbb{1}[z_j>0]. Active units pass their error back unscaled. Inactive units pass nothing, and their incoming weights get zero gradient from that example. The kink at 0 is hit with probability zero for real-valued inputs, so the subgradient choice almost never matters.

Close relatives: ReLU6 =min⁡(max⁡(0,x),6)=\min(\max(0,x),6), used in MobileNets so activations fit low-precision formats; softplus log⁡(1+ex)\log(1+e^x), a smooth version whose derivative is the sigmoid; and the leaky family (Leaky ReLU, PReLU, ELU), which give the negative side a nonzero slope.

Properties

Range
[0,∞)[0,\infty)
Monotonic
yes (non-decreasing)
Smooth
continuous, piecewise linear; not differentiable at 0
Zero-centred
no: for x∼N(0,1)x\sim\mathcal N(0,1) the mean output is 1/2π=0.3991/\sqrt{2\pi}=0.399
Saturating
not for x>0x>0 (slope 1 forever); fully flat for x<0x<0
Sparsity
exactly half the outputs are 0 for a symmetric zero-mean input
Cost
one comparison forward, one mask backward; no exp
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 1.000 · f′(x) = 1.000−1012345−4−2024x →ReLUReLU f′Leaky ReLUSoftplus
activation
compare
range [0, ∞)monotonic yessmooth nozero-centred no
Kink at 0; f′(0) is taken as 0 (the PyTorch convention).
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no sampling↑ 54%E[f]0.002.004.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.399Var[f(x)] 0.341E[f′(x)] 0.500f′ = 0 for 50.0% of inputsE[f′]¹⁰ 9.8e−4
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.ReLU and its step derivative, with Leaky ReLU and softplus for comparison. Move the probe: the slope is exactly 1 anywhere on the right and exactly 0 anywhere on the left. The output statistics show the positive mean (0.399 for standard normal inputs) and the 50% of exact zeros.

Why it trains: no saturation on the active side

Backprop multiplies one φ′(z)\varphi'(z) per layer. Sigmoid's slope is at most 0.25 and tanh's falls toward 0 for large ∣z∣|z|, so deep stacks of them shrink the gradient exponentially (see Vanishing gradients). An active ReLU contributes a factor of exactly 1, however large zz is. The only shrinkage left comes from the weights and from the units that are off, and both are handled by the initialization.

For a zero-mean symmetric pre-activation, ReLU keeps half the second moment: E[ReLU(z)2]=12Var⁡(z)\mathbb E[\mathrm{ReLU}(z)^2]=\tfrac12\operatorname{Var}(z). To keep the signal's scale constant through depth, the weights need twice the variance that a linear layer would: this is He (Kaiming) initialization.

Var⁡(z(l+1))=ninVar⁡(w) E[ReLU(z(l))2]=ninVar⁡(w)⋅12Var⁡(z(l))  ⇒  Var⁡(w)=2nin\operatorname{Var}(z^{(l+1)})=n_{in}\operatorname{Var}(w)\,\mathbb E\big[\mathrm{ReLU}(z^{(l)})^2\big]=n_{in}\operatorname{Var}(w)\cdot\tfrac12\operatorname{Var}(z^{(l)})\;\Rightarrow\;\operatorname{Var}(w)=\frac{2}{n_{in}}
ninn_{in}
fan-in of the layer
12\tfrac12
the share of E[z2]\mathbb E[z^2] that survives rectification (half the inputs are zeroed)

A ReLU network is a sum of hinges

With one hidden layer and a scalar input, each unit contributes a hinge vjmax⁡(0,wjx+bj)v_j\max(0,w_jx+b_j): zero on one side of its kink at x=−bj/wjx=-b_j/w_j and a straight line on the other. The output is their sum, so it is continuous and piecewise linear with at most one kink per unit. Training moves the kinks and the slopes until the polyline follows the data.

y^(x)=c+∑j=1Hvj max⁡(0, wjx+bj)\hat y(x)=c+\sum_{j=1}^{H} v_j\,\max(0,\,w_jx+b_j)
−bj/wj-b_j/w_j
the kink of unit jj; wj>0w_j>0 means the unit is active to the right of it
vjv_j
the hinge's output weight: how much slope the unit adds once active
HH
hidden units: at most HH kinks, so at most H+1H+1 linear pieces

Deeper ReLU networks are still piecewise linear in their input, but composition folds the space, and the number of linear regions can grow exponentially with depth (Montúfar et al., 2014). That is one formal sense in which depth buys expressive power.

ŷ = c + Σⱼ vⱼ·max(0, wⱼx + bⱼ), 8 units, 25 points−2−1012−3−2−10123▸◂▸◂▸◂▸◂unitkinkvⱼactive on1▸−2.4+0.222◂−0.1+0.133▸+0.3+0.314◂−0.1+0.105▸+1.0−0.156◂+1.7+0.227▸−1.0+0.018◂+1.1−0.26dead units: 0 / 8dead: wⱼxᵢ + bⱼ ≤ 0 for all i→ zero gradient, never revivesMSE 1.3076 · step 0101e-3━ ŷ ━ hinge, vⱼ > 0 ━ hinge, vⱼ < 0 ┄ true curve ▸ dead unit
full-batch GD, 3000 steps
fig.An 8-unit ReLU network fit to 25 noisy points by full-batch gradient descent (3000 steps). Thin lines are the individual hinges, the violet line is their sum, and ▸/◂ on the axis mark each kink and which side is active. At the default learning rate 0.03 all 8 units stay alive and the MSE reaches 0.0031. Raise the learning rate to 0.10: within the first 20 steps the large early updates push 3 units off every training point, they never come back, and the fit stalls at 0.0173. A bias shift of −3 kills 3 units before training even starts.

Worked example: three hinges

Take w=(1, −1, 0.5)w=(1,\,-1,\,0.5), b=(−0.5, 1, −1)b=(-0.5,\,1,\,-1), v=(1.2, −0.8, 2)v=(1.2,\,-0.8,\,2) and c=0.1c=0.1. The kinks are at x=0.5x=0.5, 11 and 22. On each piece, the slope of y^\hat y is the sum of vjwjv_jw_j over the active units:

x rangeactive unitsslope of ŷŷ at a point
x<0.5x<0.52(−0.8)(−1)=0.8(-0.8)(-1)=0.8y^(0)=−0.7\hat y(0)=-0.7
0.5<x<10.5<x<11, 21.2+0.8=2.01.2+0.8=2.0y^(0.75)=0.2\hat y(0.75)=0.2
1<x<21<x<211.21.2y^(1.5)=1.3\hat y(1.5)=1.3
x>2x>21, 31.2+2⋅0.5=2.21.2+2\cdot0.5=2.2y^(3)=4.1\hat y(3)=4.1
Which units are active, and the resulting slope of ŷ (checked numerically in Node)

Forward and backward at x = 1.5, target y = 1, L = ½(ŷ − y)²

  1. 1Pre-activations z=wx+b=(1.0, −0.5, −0.25)z=wx+b=(1.0,\,-0.5,\,-0.25), so h=ReLU(z)=(1.0, 0, 0)h=\mathrm{ReLU}(z)=(1.0,\,0,\,0) and y^=0.1+1.2⋅1.0=1.3\hat y=0.1+1.2\cdot1.0=1.3. Loss 0.0450.045.
  2. 2Output error ∂L/∂y^=y^−y=0.3\partial L/\partial\hat y=\hat y-y=0.3, so ∂L/∂v=0.3 h=(0.3, 0, 0)\partial L/\partial v=0.3\,h=(0.3,\,0,\,0) and ∂L/∂c=0.3\partial L/\partial c=0.3.
  3. 3Hidden errors δj=vj⋅0.3⋅1[zj>0]=(0.36, 0, 0)\delta_j=v_j\cdot0.3\cdot\mathbb 1[z_j>0]=(0.36,\,0,\,0): units 2 and 3 are off, so the gate blocks them.
  4. 4Weight gradients ∂L/∂wj=δjx=(0.54, 0, 0)\partial L/\partial w_j=\delta_j x=(0.54,\,0,\,0) and ∂L/∂bj=δj=(0.36, 0, 0)\partial L/\partial b_j=\delta_j=(0.36,\,0,\,0). Only unit 1 learns from this example.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-2.0, -0.5, 0.0, 0.5, 2.0], requires_grad=True)
4
5y = nn.ReLU()(x) # module, for nn.Sequential
6y = F.relu(x) # functional; F.relu_(t) / nn.ReLU(inplace=True) overwrite t
7y.sum().backward()
8print(y.tolist()) # [0.0, 0.0, 0.0, 0.5, 2.0]
9print(x.grad.tolist()) # [0.0, 0.0, 0.0, 1.0, 1.0] <- f'(0) is taken as 0
10
11# From scratch: forward is a clamp, backward is a mask
12class MyReLU(torch.autograd.Function):
13 @staticmethod
14 def forward(ctx, x):
15 ctx.save_for_backward(x)
16 return x.clamp(min=0)
17 @staticmethod
18 def backward(ctx, g):
19 (x,) = ctx.saved_tensors
20 return g * (x > 0).to(g.dtype)
21
22# Dead units: zero on every example of a (large) batch
23lin = nn.Linear(64, 256)
24nn.init.kaiming_normal_(lin.weight, nonlinearity="relu") # Var(w) = 2 / fan_in
25z = lin(torch.randn(4096, 64))
26dead = (z <= 0).all(dim=0)
27print(f"dead units: {dead.sum().item()} / {z.shape[1]}")
ReLU three ways, plus a dead-unit check

Use when

  • +Gradient exactly 1 on the active side: no saturation, healthy gradient flow in deep nets
  • +Cheapest nonlinearity: a comparison forward, a mask backward
  • +Sparse activations: about half the units are zero for a given input
  • +Piecewise linear: easy to analyse, and compatible with He init

Avoid when

  • −Not zero-centred: mean output 0.399σ0.399\sigma shifts the next layer's inputs
  • −Dying units: zero gradient for all inputs once a unit is off everywhere
  • −Not differentiable at 0, and the kinks make the loss surface non-smooth
  • −Unbounded output: activations can grow without normalization

Check yourself

A ReLU unit has z ≤ 0 for every example in the training set. What happens to its incoming weights under gradient descent?

Check yourself

A 1-input network has one hidden layer of 8 ReLU units and a linear output. At most how many linear pieces can its output have?

Remember

  • ✓ReLU(x)=max⁡(0,x)\mathrm{ReLU}(x)=\max(0,x), ReLU′(x)=1[x>0]\mathrm{ReLU}'(x)=\mathbb 1[x>0] (frameworks use 0 at the kink).
  • ✓Active units pass gradient with factor exactly 1, so ReLU avoids the saturation that made deep sigmoid/tanh nets hard to train.
  • ✓Pair it with He init, Var⁡(w)=2/nin\operatorname{Var}(w)=2/n_{in}, because rectification keeps half the second moment.
  • ✓A one-hidden-layer ReLU net is a sum of hinges: continuous and piecewise linear, at most H+1H+1 pieces in 1-D.
  • ✓Dying ReLU: a unit off on every input never gets gradient again; too-high learning rates and negative biases cause it.
← back to Deep Learning roadmap