Leaky ReLU

max(αx, x) with a small fixed α such as 0.01: the negative side keeps slope α, so no unit ever has zero gradient and a dead ReLU unit can climb back.

8 min read678 wordscore0 viewsupdated 2026-10-11

Definition

Leaky ReLU is the activation LeakyReLUα(x)=max⁡(αx, x)\mathrm{LeakyReLU}_\alpha(x)=\max(\alpha x,\,x) for a fixed slope 0<α<10<\alpha<1. It equals ReLU for x>0x>0 and lets a small multiple αx\alpha x "leak" through for x<0x<0, so its derivative is never zero.

ReLU's weakness is the flat half. A unit whose pre-activation is negative on every input gets exactly zero gradient and stays dead. Leaky ReLU removes the flat part with the smallest possible change: a straight line of slope α\alpha on the left. Everything else, the cost, the piecewise linearity, the identity slope on the right, stays the same.

Maas, Hannun and Ng introduced it in 2013 ("Rectifier nonlinearities improve neural network acoustic models") with α=0.01\alpha=0.01, which is still PyTorch's default. In their speech-recognition experiments rectifier networks clearly beat tanh, and leaky and plain ReLU performed about the same. Its lasting role is as the safe variant where dead units would hurt, above all in GAN discriminators.

Formula and derivative

LeakyReLUα(x)={xx>0αxx≤0LeakyReLUα′(x)={1x>0αx<0\mathrm{LeakyReLU}_\alpha(x)=\begin{cases}x & x>0\\ \alpha x & x\le 0\end{cases}\qquad\qquad \mathrm{LeakyReLU}_\alpha'(x)=\begin{cases}1 & x>0\\ \alpha & x<0\end{cases}
α\alpha
the negative slope, a fixed hyperparameter: 0.01 (Maas et al.), 0.1 (YOLO), 0.2 (DCGAN)
max⁡(αx,x)\max(\alpha x,x)
a valid one-line form only for α≤1\alpha\le1
f′(0)f'(0)
undefined; PyTorch returns α\alpha at exactly 0

α=0\alpha=0 gives ReLU and α=1\alpha=1 the identity, so α\alpha interpolates between a sparse gate and a linear layer. Learning α\alpha per channel gives PReLU. Drawing it at random in training (uniform on [1/8,1/3][1/8,1/3] in PyTorch's nn.RReLU) and fixing it to the mean at test time gives RReLU (Xu et al., 2015).

Properties

Range
(−∞,∞)(-\infty,\infty)
Monotonic
yes, strictly increasing for α>0\alpha>0
Smooth
continuous, piecewise linear; kink at 0 for every α≠1\alpha\ne1
Zero-centred
no, but closer than ReLU: mean output for N(0,1)\mathcal N(0,1) input is 0.3950.395 at α=0.01\alpha=0.01 and 0.3190.319 at α=0.2\alpha=0.2 (ReLU 0.3990.399)
Saturating
no: slope 1 on the right, α on the left, everywhere
Gradient
never zero: min⁡f′=α\min f'=\alpha, so no unit can die
Cost
one comparison and one multiply; no exp
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 1.000 · f′(x) = 1.000−1012345−4−2024x →Leaky ReLU (α=0.01)Leaky ReLU f′ReLU
activation
compare
range (−∞, ∞)monotonic yessmooth nozero-centred no
Fixed negative slope α (default 0.01). f′(0) is taken as α.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no sampling↑ 53%E[f]−0.041.984.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.395Var[f(x)] 0.344E[f′(x)] 0.505f′ = 0 for 0.0% of inputsE[f′]¹⁰ 1.1e−3
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.Leaky ReLU against ReLU. Drag α up from 0.01: the left branch tilts and the derivative's left level rises from 0 to α. The output mean falls a little and the share of exactly-zero gradients drops from 50% to 0.

Fixing dead units

The backpropagated error at a unit is δj=(W⊤δ(l+1))j⋅f′(zj)\delta_j=(W^\top\delta^{(l+1)})_j\cdot f'(z_j). With ReLU, a unit that is off on every example multiplies every error by 0. With Leaky ReLU it multiplies them by α\alpha, so its weights and bias still get a gradient pointing in whatever direction lowers the loss. If switching the unit back on would help, gradient descent will move it there.

∂L∂bj=∑i∂L∂hij f′(zij)=α∑i∂L∂hijif zij<0 for every example i\frac{\partial\mathcal L}{\partial b_j}=\sum_{i}\frac{\partial\mathcal L}{\partial h_{ij}}\,f'(z_{ij})=\alpha\sum_{i}\frac{\partial\mathcal L}{\partial h_{ij}}\qquad\text{if } z_{ij}<0\text{ for every example } i
hijh_{ij}
output of unit jj on example ii
α∑i⋯\alpha\sum_i\cdots
ReLU gives exactly 0 here; Leaky ReLU gives a signal scaled by α\alpha, so recovery takes about 1/α1/\alpha times longer than a normal update
mean f′(z) per unit (log scale) · 48 He-init units, batch 200, bias shift −3.0ReLU10.10.01✕✕✕✕✕leaky10.10.01ReLU: 5 dead (f′ = 0 on all 200), mean f′ 2.0% · leaky α=0.01: 0 dead, mean f′ 3.0%bias b of one unit fit to max(0, x), per GD step10−1−2−3off on every input (w = 1)0100200300400the unit at step 0x ∈ [−2, 2]━ ReLU b = −3.00, MSE 0.7333, never wakes━ leaky b = −3.00, MSE 0.7511, wakes at step 51
fan-in 16, lr 0.5, 400 steps
fig.Top: 48 He-initialised units on a batch of 200 inputs, all biases shifted by the slider; bars are each unit's mean f′(z) (log scale), the factor by which errors reach its weights. At shift −3, ReLU has 5 dead units (✕) and a mean slope of 2.0%; leaky (α = 0.01) has none and 3.0%. At −4 ReLU loses 29 of 48. Bottom: one unit fit to y = max(0, x) from b₀ = shift. Press run GD at shift −3: the ReLU unit never moves (MSE stays 0.7333), the leaky one wakes at step 51 and ends at b = 0.001, MSE 0.00007. At α = 0.1 it wakes at step 6, but ends with a worse fit (MSE 0.0066), because the target's flat side has slope 0, not α.

Initialization and the choice of α

For a symmetric zero-mean input, Leaky ReLU keeps E[f(z)2]=1+α22Var⁡(z)\mathbb E[f(z)^2]=\tfrac{1+\alpha^2}{2}\operatorname{Var}(z), so the He condition becomes Var⁡(w)=2/((1+α2) nin)\operatorname{Var}(w)=2/\big((1+\alpha^2)\,n_{in}\big). This is PyTorch's calculate_gain("leaky_relu", a) =2/(1+a2)=\sqrt{2/(1+a^2)}: 1.41411.4141 at α=0.01\alpha=0.01 (indistinguishable from ReLU's 2\sqrt2) and 1.38681.3868 at α=0.2\alpha=0.2.

Small α\alpha keeps ReLU's sparsity and behaviour while guaranteeing a gradient. Larger α\alpha (0.1 to 0.3) gives a stronger gradient on the left and a more symmetric output, at the cost of a more linear unit: at α=1\alpha=1 the nonlinearity is gone. Xu et al. (2015) found larger slopes such as 1/5.5≈0.181/5.5\approx0.18 competitive with ReLU on CIFAR. In practice α\alpha is tuned like any hyperparameter, or learned (PReLU).

Worked example

xReLUReLU′leaky α=0.01f′leaky α=0.2f′
−300−0.030.01−0.60.2
−100−0.010.01−0.20.2
−0.500−0.0050.01−0.10.2
0.50.510.510.51
2212121
f and f′ for ReLU and two leaky slopes (computed in Node)

First gradient step for the dead unit in the figure (w = 1, b = −3, lr = 0.5)

  1. 1On the 21 inputs x∈[−2,2]x\in[-2,2], z=x−3∈[−5,−1]z=x-3\in[-5,-1]: every input is on the left of the kink.
  2. 2ReLU: f′(z)=0f'(z)=0 on all 21, so ∂L/∂w=∂L/∂b=0\partial L/\partial w=\partial L/\partial b=0. The unit is frozen at MSE 0.73330.7333 forever.
  3. 3Leaky, α=0.01\alpha=0.01: the residuals αz−y\alpha z-y are all negative (the unit outputs about −0.03-0.03 where the target is up to 2), so both gradients are negative. The first step moves bb by +0.0055+0.0055 and ww by +0.0072+0.0072.
  4. 4Those small steps compound: by step 51 the largest input x=2x=2 has wx+b>0wx+b>0, the unit is back on the steep side, and it then converges like a normal ReLU unit. With α=0.1\alpha=0.1 the first step is +0.082+0.082 and the unit wakes at step 6.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-3.0, -1.0, -0.5, 0.5, 2.0], requires_grad=True)
4
5y = nn.LeakyReLU(negative_slope=0.01)(x) # module (default slope 0.01)
6y = F.leaky_relu(x, negative_slope=0.01) # functional
7y.sum().backward()
8print(y.detach()) # tensor([-0.0300, -0.0100, -0.0050, 0.5000, 2.0000])
9print(x.grad) # tensor([0.0100, 0.0100, 0.0100, 1.0000, 1.0000]) <- never zero
10
11def leaky_relu(x, alpha=0.01): # from scratch; autograd differentiates torch.where
12 return torch.where(x > 0, x, alpha * x)
13
14# He / Kaiming init with the leaky gain sqrt(2 / (1 + a^2))
15lin = nn.Linear(256, 256)
16nn.init.kaiming_normal_(lin.weight, a=0.2, nonlinearity="leaky_relu")
17print(nn.init.calculate_gain("leaky_relu", 0.2)) # 1.3867504905630728
Leaky ReLU three ways, and He init that accounts for α

Where it's used

GAN discriminators

DCGAN and most of its descendants use slope 0.2, so gradients reach the generator through every unit.

Object detection

YOLO's Darknet backbones use leaky units with slope 0.1.

Speech and small models

The original 0.01 variant from acoustic modelling; a cheap safety margin wherever dead units show up.

Use when

  • +No dead units: the gradient is at least α everywhere
  • +Same cost as ReLU and the same identity slope on the right
  • +Slightly closer to zero-centred than ReLU
  • +Drop-in replacement; He init needs only the gain √(2/(1+α²))

Avoid when

  • −α is one more hyperparameter, and results can depend on it
  • −Small α revives units only slowly
  • −Large α makes the unit more linear and less sparse
  • −Still unbounded, kinked at 0, and not zero-centred

Check yourself

A leaky ReLU unit with α = 0.01 has z < 0 on every training example. Compared with an active unit receiving the same upstream errors, its bias gradient is…

Check yourself

Which He-initialization standard deviation keeps the forward variance constant for Leaky ReLU with slope α?

Remember

  • ✓LeakyReLUα(x)=max⁡(αx,x)\mathrm{LeakyReLU}_\alpha(x)=\max(\alpha x,x); derivative 1 for x>0x>0, α\alpha for x<0x<0. Default α=0.01\alpha=0.01 (Maas et al., 2013).
  • ✓The gradient is never zero, so units cannot die; a unit stuck on the left learns at about α\alpha times the normal rate.
  • ✓He init gain becomes 2/(1+α2)\sqrt{2/(1+\alpha^2)}; for α=0.01\alpha=0.01 that is essentially ReLU's 2\sqrt2.
  • ✓Standard in GAN discriminators (DCGAN, slope 0.2) and YOLO backbones (slope 0.1).
  • ✓Learned α is PReLU; random α in training is RReLU.
← back to Deep Learning roadmap