Definition
ReLU (rectified linear unit) is the activation . It passes positive pre-activations through unchanged and sets negative ones to zero, so a unit is either on (slope 1) or off (output and slope 0).
ReLU is the default hidden activation of modern deep learning. It replaced sigmoid and tanh because it does not saturate for positive inputs, costs one comparison, and makes many units exactly zero. It is also the simplest nonlinearity that still gives a network its expressive power.
The function is old. Fukushima used rectifying units in his 1969 visual feature extractor and in the cognitron. The modern revival came in three steps. Nair and Hinton (2010) used rectified linear units in restricted Boltzmann machines. Glorot, Bordes and Bengio (2011) showed that deep rectifier networks train well without unsupervised pre-training and produce sparse representations. Krizhevsky, Sutskever and Hinton's AlexNet (2012) reported that a 4-layer CNN with ReLUs reached 25% training error on CIFAR-10 six times faster than the same network with tanh.
Formula and derivative
- the unit's pre-activation
- undefined; any value in is a valid subgradient. PyTorch, TensorFlow and JAX use 0
- the Heaviside step , a gate that either passes the backpropagated error or blocks it
In backprop the derivative acts as a mask: . Active units pass their error back unscaled. Inactive units pass nothing, and their incoming weights get zero gradient from that example. The kink at 0 is hit with probability zero for real-valued inputs, so the subgradient choice almost never matters.
Close relatives: ReLU6 , used in MobileNets so activations fit low-precision formats; softplus , a smooth version whose derivative is the sigmoid; and the leaky family (Leaky ReLU, PReLU, ELU), which give the negative side a nonzero slope.
Properties
- Range
- Monotonic
- yes (non-decreasing)
- Smooth
- continuous, piecewise linear; not differentiable at 0
- Zero-centred
- no: for the mean output is
- Saturating
- not for (slope 1 forever); fully flat for
- Sparsity
- exactly half the outputs are 0 for a symmetric zero-mean input
- Cost
- one comparison forward, one mask backward; no exp
Why it trains: no saturation on the active side
Backprop multiplies one per layer. Sigmoid's slope is at most 0.25 and tanh's falls toward 0 for large , so deep stacks of them shrink the gradient exponentially (see Vanishing gradients). An active ReLU contributes a factor of exactly 1, however large is. The only shrinkage left comes from the weights and from the units that are off, and both are handled by the initialization.
For a zero-mean symmetric pre-activation, ReLU keeps half the second moment: . To keep the signal's scale constant through depth, the weights need twice the variance that a linear layer would: this is He (Kaiming) initialization.
- fan-in of the layer
- the share of that survives rectification (half the inputs are zeroed)
A ReLU network is a sum of hinges
With one hidden layer and a scalar input, each unit contributes a hinge : zero on one side of its kink at and a straight line on the other. The output is their sum, so it is continuous and piecewise linear with at most one kink per unit. Training moves the kinks and the slopes until the polyline follows the data.
- the kink of unit ; means the unit is active to the right of it
- the hinge's output weight: how much slope the unit adds once active
- hidden units: at most kinks, so at most linear pieces
Deeper ReLU networks are still piecewise linear in their input, but composition folds the space, and the number of linear regions can grow exponentially with depth (Montúfar et al., 2014). That is one formal sense in which depth buys expressive power.
Worked example: three hinges
Take , , and . The kinks are at , and . On each piece, the slope of is the sum of over the active units:
| x range | active units | slope of ŷ | ŷ at a point |
|---|---|---|---|
| 2 | |||
| 1, 2 | |||
| 1 | |||
| 1, 3 |
Forward and backward at x = 1.5, target y = 1, L = ½(ŷ − y)²
- 1Pre-activations , so and . Loss .
- 2Output error , so and .
- 3Hidden errors : units 2 and 3 are off, so the gate blocks them.
- 4Weight gradients and . Only unit 1 learns from this example.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([-2.0, -0.5, 0.0, 0.5, 2.0], requires_grad=True)45y = nn.ReLU()(x) # module, for nn.Sequential6y = F.relu(x) # functional; F.relu_(t) / nn.ReLU(inplace=True) overwrite t7y.sum().backward()8print(y.tolist()) # [0.0, 0.0, 0.0, 0.5, 2.0]9print(x.grad.tolist()) # [0.0, 0.0, 0.0, 1.0, 1.0] <- f'(0) is taken as 01011# From scratch: forward is a clamp, backward is a mask12class MyReLU(torch.autograd.Function):13 @staticmethod14 def forward(ctx, x):15 ctx.save_for_backward(x)16 return x.clamp(min=0)17 @staticmethod18 def backward(ctx, g):19 (x,) = ctx.saved_tensors20 return g * (x > 0).to(g.dtype)2122# Dead units: zero on every example of a (large) batch23lin = nn.Linear(64, 256)24nn.init.kaiming_normal_(lin.weight, nonlinearity="relu") # Var(w) = 2 / fan_in25z = lin(torch.randn(4096, 64))26dead = (z <= 0).all(dim=0)27print(f"dead units: {dead.sum().item()} / {z.shape[1]}")Use when
- +Gradient exactly 1 on the active side: no saturation, healthy gradient flow in deep nets
- +Cheapest nonlinearity: a comparison forward, a mask backward
- +Sparse activations: about half the units are zero for a given input
- +Piecewise linear: easy to analyse, and compatible with He init
Avoid when
- −Not zero-centred: mean output shifts the next layer's inputs
- −Dying units: zero gradient for all inputs once a unit is off everywhere
- −Not differentiable at 0, and the kinks make the loss surface non-smooth
- −Unbounded output: activations can grow without normalization
Check yourself
A ReLU unit has z ≤ 0 for every example in the training set. What happens to its incoming weights under gradient descent?
Check yourself
A 1-input network has one hidden layer of 8 ReLU units and a linear output. At most how many linear pieces can its output have?
Remember
- ✓, (frameworks use 0 at the kink).
- ✓Active units pass gradient with factor exactly 1, so ReLU avoids the saturation that made deep sigmoid/tanh nets hard to train.
- ✓Pair it with He init, , because rectification keeps half the second moment.
- ✓A one-hidden-layer ReLU net is a sum of hinges: continuous and piecewise linear, at most pieces in 1-D.
- ✓Dying ReLU: a unit off on every input never gets gradient again; too-high learning rates and negative biases cause it.