Definition
Leaky ReLU is the activation for a fixed slope . It equals ReLU for and lets a small multiple "leak" through for , so its derivative is never zero.
ReLU's weakness is the flat half. A unit whose pre-activation is negative on every input gets exactly zero gradient and stays dead. Leaky ReLU removes the flat part with the smallest possible change: a straight line of slope on the left. Everything else, the cost, the piecewise linearity, the identity slope on the right, stays the same.
Maas, Hannun and Ng introduced it in 2013 ("Rectifier nonlinearities improve neural network acoustic models") with , which is still PyTorch's default. In their speech-recognition experiments rectifier networks clearly beat tanh, and leaky and plain ReLU performed about the same. Its lasting role is as the safe variant where dead units would hurt, above all in GAN discriminators.
Formula and derivative
- the negative slope, a fixed hyperparameter: 0.01 (Maas et al.), 0.1 (YOLO), 0.2 (DCGAN)
- a valid one-line form only for
- undefined; PyTorch returns at exactly 0
gives ReLU and the identity, so interpolates between a sparse gate and a linear layer. Learning per channel gives PReLU. Drawing it at random in training (uniform on in PyTorch's nn.RReLU) and fixing it to the mean at test time gives RReLU (Xu et al., 2015).
Properties
- Range
- Monotonic
- yes, strictly increasing for
- Smooth
- continuous, piecewise linear; kink at 0 for every
- Zero-centred
- no, but closer than ReLU: mean output for input is at and at (ReLU )
- Saturating
- no: slope 1 on the right, α on the left, everywhere
- Gradient
- never zero: , so no unit can die
- Cost
- one comparison and one multiply; no exp
Fixing dead units
The backpropagated error at a unit is . With ReLU, a unit that is off on every example multiplies every error by 0. With Leaky ReLU it multiplies them by , so its weights and bias still get a gradient pointing in whatever direction lowers the loss. If switching the unit back on would help, gradient descent will move it there.
- output of unit on example
- ReLU gives exactly 0 here; Leaky ReLU gives a signal scaled by , so recovery takes about times longer than a normal update
Initialization and the choice of α
For a symmetric zero-mean input, Leaky ReLU keeps , so the He condition becomes . This is PyTorch's calculate_gain("leaky_relu", a) : at (indistinguishable from ReLU's ) and at .
Small keeps ReLU's sparsity and behaviour while guaranteeing a gradient. Larger (0.1 to 0.3) gives a stronger gradient on the left and a more symmetric output, at the cost of a more linear unit: at the nonlinearity is gone. Xu et al. (2015) found larger slopes such as competitive with ReLU on CIFAR. In practice is tuned like any hyperparameter, or learned (PReLU).
Worked example
| x | ReLU | ReLU′ | leaky α=0.01 | f′ | leaky α=0.2 | f′ |
|---|---|---|---|---|---|---|
| −3 | 0 | 0 | −0.03 | 0.01 | −0.6 | 0.2 |
| −1 | 0 | 0 | −0.01 | 0.01 | −0.2 | 0.2 |
| −0.5 | 0 | 0 | −0.005 | 0.01 | −0.1 | 0.2 |
| 0.5 | 0.5 | 1 | 0.5 | 1 | 0.5 | 1 |
| 2 | 2 | 1 | 2 | 1 | 2 | 1 |
First gradient step for the dead unit in the figure (w = 1, b = −3, lr = 0.5)
- 1On the 21 inputs , : every input is on the left of the kink.
- 2ReLU: on all 21, so . The unit is frozen at MSE forever.
- 3Leaky, : the residuals are all negative (the unit outputs about where the target is up to 2), so both gradients are negative. The first step moves by and by .
- 4Those small steps compound: by step 51 the largest input has , the unit is back on the steep side, and it then converges like a normal ReLU unit. With the first step is and the unit wakes at step 6.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([-3.0, -1.0, -0.5, 0.5, 2.0], requires_grad=True)45y = nn.LeakyReLU(negative_slope=0.01)(x) # module (default slope 0.01)6y = F.leaky_relu(x, negative_slope=0.01) # functional7y.sum().backward()8print(y.detach()) # tensor([-0.0300, -0.0100, -0.0050, 0.5000, 2.0000])9print(x.grad) # tensor([0.0100, 0.0100, 0.0100, 1.0000, 1.0000]) <- never zero1011def leaky_relu(x, alpha=0.01): # from scratch; autograd differentiates torch.where12 return torch.where(x > 0, x, alpha * x)1314# He / Kaiming init with the leaky gain sqrt(2 / (1 + a^2))15lin = nn.Linear(256, 256)16nn.init.kaiming_normal_(lin.weight, a=0.2, nonlinearity="leaky_relu")17print(nn.init.calculate_gain("leaky_relu", 0.2)) # 1.3867504905630728Where it's used
GAN discriminators
DCGAN and most of its descendants use slope 0.2, so gradients reach the generator through every unit.
Object detection
YOLO's Darknet backbones use leaky units with slope 0.1.
Speech and small models
The original 0.01 variant from acoustic modelling; a cheap safety margin wherever dead units show up.
Use when
- +No dead units: the gradient is at least α everywhere
- +Same cost as ReLU and the same identity slope on the right
- +Slightly closer to zero-centred than ReLU
- +Drop-in replacement; He init needs only the gain √(2/(1+α²))
Avoid when
- −α is one more hyperparameter, and results can depend on it
- −Small α revives units only slowly
- −Large α makes the unit more linear and less sparse
- −Still unbounded, kinked at 0, and not zero-centred
Check yourself
A leaky ReLU unit with α = 0.01 has z < 0 on every training example. Compared with an active unit receiving the same upstream errors, its bias gradient is…
Check yourself
Which He-initialization standard deviation keeps the forward variance constant for Leaky ReLU with slope α?
Remember
- ✓; derivative 1 for , for . Default (Maas et al., 2013).
- ✓The gradient is never zero, so units cannot die; a unit stuck on the left learns at about times the normal rate.
- ✓He init gain becomes ; for that is essentially ReLU's .
- ✓Standard in GAN discriminators (DCGAN, slope 0.2) and YOLO backbones (slope 0.1).
- ✓Learned α is PReLU; random α in training is RReLU.