# Leaky ReLU

> max(αx, x) with a small fixed α such as 0.01: the negative side keeps slope α, so no unit ever has zero gradient and a dead ReLU unit can climb back.

Source: https://www.mlfoundry.in/learn/leaky-relu  
Section: Deep Learning › Activation Functions  
Level: core · 8 min read · updated 2026-10-11

**Leaky ReLU:** is the activation $\mathrm{LeakyReLU}_\alpha(x)=\max(\alpha x,\,x)$ for a fixed slope $0<\alpha<1$. It equals ReLU for $x>0$ and lets a small multiple $\alpha x$ "leak" through for $x<0$, so its derivative is never zero.

ReLU's weakness is the flat half. A unit whose pre-activation is negative on every input gets exactly zero gradient and stays dead. Leaky ReLU removes the flat part with the smallest possible change: a straight line of slope $\alpha$ on the left. Everything else, the cost, the piecewise linearity, the identity slope on the right, stays the same.

Maas, Hannun and Ng introduced it in 2013 ("Rectifier nonlinearities improve neural network acoustic models") with $\alpha=0.01$, which is still PyTorch's default. In their speech-recognition experiments rectifier networks clearly beat tanh, and leaky and plain ReLU performed about the same. Its lasting role is as the safe variant where dead units would hurt, above all in GAN discriminators.

## Formula and derivative

$$
\mathrm{LeakyReLU}_\alpha(x)=\begin{cases}x & x>0\\ \alpha x & x\le 0\end{cases}\qquad\qquad \mathrm{LeakyReLU}_\alpha'(x)=\begin{cases}1 & x>0\\ \alpha & x<0\end{cases}
$$

- $\alpha$: the negative slope, a fixed hyperparameter: 0.01 (Maas et al.), 0.1 (YOLO), 0.2 (DCGAN)
- $\max(\alpha x,x)$: a valid one-line form only for $\alpha\le1$
- $f'(0)$: undefined; PyTorch returns $\alpha$ at exactly 0

$\alpha=0$ gives ReLU and $\alpha=1$ the identity, so $\alpha$ interpolates between a sparse gate and a linear layer. Learning $\alpha$ per channel gives **PReLU**. Drawing it at random in training (uniform on $[1/8,1/3]$ in PyTorch's `nn.RReLU`) and fixing it to the mean at test time gives **RReLU** (Xu et al., 2015).

**Properties**

- **Range:** $(-\infty,\infty)$
- **Monotonic:** yes, strictly increasing for $\alpha>0$
- **Smooth:** continuous, piecewise linear; kink at 0 for every $\alpha\ne1$
- **Zero-centred:** no, but closer than ReLU: mean output for $\mathcal N(0,1)$ input is $0.395$ at $\alpha=0.01$ and $0.319$ at $\alpha=0.2$ (ReLU $0.399$)
- **Saturating:** no: slope 1 on the right, α on the left, everywhere
- **Gradient:** never zero: $\min f'=\alpha$, so **no unit can die**
- **Cost:** one comparison and one multiply; no exp

> Figure: Leaky ReLU against ReLU. Drag α up from 0.01: the left branch tilts and the derivative's left level rises from 0 to α. The output mean falls a little and the share of exactly-zero gradients drops from 50% to 0.

## Fixing dead units

The backpropagated error at a unit is $\delta_j=(W^\top\delta^{(l+1)})_j\cdot f'(z_j)$. With ReLU, a unit that is off on every example multiplies every error by 0. With Leaky ReLU it multiplies them by $\alpha$, so its weights and bias still get a gradient pointing in whatever direction lowers the loss. If switching the unit back on would help, gradient descent will move it there.

$$
\frac{\partial\mathcal L}{\partial b_j}=\sum_{i}\frac{\partial\mathcal L}{\partial h_{ij}}\,f'(z_{ij})=\alpha\sum_{i}\frac{\partial\mathcal L}{\partial h_{ij}}\qquad\text{if } z_{ij}<0\text{ for every example } i
$$

- $h_{ij}$: output of unit $j$ on example $i$
- $\alpha\sum_i\cdots$: ReLU gives exactly 0 here; Leaky ReLU gives a signal scaled by $\alpha$, so recovery takes about $1/\alpha$ times longer than a normal update

> Figure: Top: 48 He-initialised units on a batch of 200 inputs, all biases shifted by the slider; bars are each unit's mean f′(z) (log scale), the factor by which errors reach its weights. At shift −3, ReLU has 5 dead units (✕) and a mean slope of 2.0%; leaky (α = 0.01) has none and 3.0%. At −4 ReLU loses 29 of 48. Bottom: one unit fit to y = max(0, x) from b₀ = shift. Press run GD at shift −3: the ReLU unit never moves (MSE stays 0.7333), the leaky one wakes at step 51 and ends at b = 0.001, MSE 0.00007. At α = 0.1 it wakes at step 6, but ends with a worse fit (MSE 0.0066), because the target's flat side has slope 0, not α.

## Initialization and the choice of α

For a symmetric zero-mean input, Leaky ReLU keeps $\mathbb E[f(z)^2]=\tfrac{1+\alpha^2}{2}\operatorname{Var}(z)$, so the He condition becomes $\operatorname{Var}(w)=2/\big((1+\alpha^2)\,n_{in}\big)$. This is PyTorch's `calculate_gain("leaky_relu", a)` $=\sqrt{2/(1+a^2)}$: $1.4141$ at $\alpha=0.01$ (indistinguishable from ReLU's $\sqrt2$) and $1.3868$ at $\alpha=0.2$.

Small $\alpha$ keeps ReLU's sparsity and behaviour while guaranteeing a gradient. Larger $\alpha$ (0.1 to 0.3) gives a stronger gradient on the left and a more symmetric output, at the cost of a more linear unit: at $\alpha=1$ the nonlinearity is gone. Xu et al. (2015) found larger slopes such as $1/5.5\approx0.18$ competitive with ReLU on CIFAR. In practice $\alpha$ is tuned like any hyperparameter, or learned (PReLU).

## Worked example

f and f′ for ReLU and two leaky slopes (computed in Node)

| x | ReLU | ReLU′ | leaky α=0.01 | f′ | leaky α=0.2 | f′ |
| --- | --- | --- | --- | --- | --- | --- |
| −3 | 0 | 0 | −0.03 | 0.01 | −0.6 | 0.2 |
| −1 | 0 | 0 | −0.01 | 0.01 | −0.2 | 0.2 |
| −0.5 | 0 | 0 | −0.005 | 0.01 | −0.1 | 0.2 |
| 0.5 | 0.5 | 1 | 0.5 | 1 | 0.5 | 1 |
| 2 | 2 | 1 | 2 | 1 | 2 | 1 |

**First gradient step for the dead unit in the figure (w = 1, b = −3, lr = 0.5)**

1. On the 21 inputs $x\in[-2,2]$, $z=x-3\in[-5,-1]$: every input is on the left of the kink.
2. ReLU: $f'(z)=0$ on all 21, so $\partial L/\partial w=\partial L/\partial b=0$. The unit is frozen at MSE $0.7333$ forever.
3. Leaky, $\alpha=0.01$: the residuals $\alpha z-y$ are all negative (the unit outputs about $-0.03$ where the target is up to 2), so both gradients are negative. The first step moves $b$ by $+0.0055$ and $w$ by $+0.0072$.
4. Those small steps compound: by step 51 the largest input $x=2$ has $wx+b>0$, the unit is back on the steep side, and it then converges like a normal ReLU unit. With $\alpha=0.1$ the first step is $+0.082$ and the unit wakes at step 6.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-3.0, -1.0, -0.5, 0.5, 2.0], requires_grad=True)

y = nn.LeakyReLU(negative_slope=0.01)(x)     # module (default slope 0.01)
y = F.leaky_relu(x, negative_slope=0.01)     # functional
y.sum().backward()
print(y.detach())        # tensor([-0.0300, -0.0100, -0.0050,  0.5000,  2.0000])
print(x.grad)            # tensor([0.0100, 0.0100, 0.0100, 1.0000, 1.0000])  <- never zero

def leaky_relu(x, alpha=0.01):               # from scratch; autograd differentiates torch.where
    return torch.where(x > 0, x, alpha * x)

# He / Kaiming init with the leaky gain sqrt(2 / (1 + a^2))
lin = nn.Linear(256, 256)
nn.init.kaiming_normal_(lin.weight, a=0.2, nonlinearity="leaky_relu")
print(nn.init.calculate_gain("leaky_relu", 0.2))   # 1.3867504905630728
```

Leaky ReLU three ways, and He init that accounts for α

> **Not a cure for every ReLU problem:** Leaky ReLU guarantees a gradient, not a good one. A unit stuck far on the left still learns at rate $\alpha$ times normal, so with $\alpha=0.01$ revival can take hundreds of steps, and the root cause (a learning rate that is too high, a bad init) is still there. It also keeps ReLU's unbounded positive side and its non-zero mean.

> **Why GAN discriminators leak:** The generator learns only through the gradient of the discriminator's output with respect to its **input**. That gradient passes through every discriminator activation, and with ReLU every off unit blocks it for that sample. DCGAN (Radford, Metz and Chintala, 2015) used Leaky ReLU with slope 0.2 in all discriminator layers (and ReLU in the generator), and that choice became standard in GAN architectures.

- **GAN discriminators:** DCGAN and most of its descendants use slope 0.2, so gradients reach the generator through every unit.
- **Object detection:** YOLO's Darknet backbones use leaky units with slope 0.1.
- **Speech and small models:** The original 0.01 variant from acoustic modelling; a cheap safety margin wherever dead units show up.

**Strengths**

- No dead units: the gradient is at least α everywhere
- Same cost as ReLU and the same identity slope on the right
- Slightly closer to zero-centred than ReLU
- Drop-in replacement; He init needs only the gain √(2/(1+α²))

**Limitations**

- α is one more hyperparameter, and results can depend on it
- Small α revives units only slowly
- Large α makes the unit more linear and less sparse
- Still unbounded, kinked at 0, and not zero-centred

**Question:** A leaky ReLU unit with α = 0.01 has z < 0 on every training example. Compared with an active unit receiving the same upstream errors, its bias gradient is…

- A. Zero, as for ReLU
- B. Scaled by 0.01
- C. Scaled by 0.5
- D. The same, because the slope does not affect the bias

**Answer:** B. Scaled by 0.01 — $\partial L/\partial b_j=\sum_i(\partial L/\partial h_{ij})\,f'(z_{ij})$ and $f'=\alpha=0.01$ on every example. The direction is the same as for an active unit, the size is 100× smaller, which is why revival is slow for small $\alpha$.

**Question:** Which He-initialization standard deviation keeps the forward variance constant for Leaky ReLU with slope α?

- A. √(2 / n_in)
- B. √(2 / ((1 + α²) n_in))
- C. √(1 / n_in)
- D. √(2(1 + α²) / n_in)

**Answer:** B. √(2 / ((1 + α²) n_in)) — For symmetric zero-mean $z$, $\mathbb E[f(z)^2]=\tfrac{1+\alpha^2}{2}\operatorname{Var}(z)$. Setting $n_{in}\operatorname{Var}(w)\cdot\tfrac{1+\alpha^2}{2}=1$ gives $\operatorname{Var}(w)=2/((1+\alpha^2)n_{in})$, which reduces to He's $2/n_{in}$ at $\alpha=0$.

**Key points**

- $\mathrm{LeakyReLU}_\alpha(x)=\max(\alpha x,x)$; derivative 1 for $x>0$, $\alpha$ for $x<0$. Default $\alpha=0.01$ (Maas et al., 2013).
- The gradient is never zero, so units cannot die; a unit stuck on the left learns at about $\alpha$ times the normal rate.
- He init gain becomes $\sqrt{2/(1+\alpha^2)}$; for $\alpha=0.01$ that is essentially ReLU's $\sqrt2$.
- Standard in GAN discriminators (DCGAN, slope 0.2) and YOLO backbones (slope 0.1).
- Learned α is PReLU; random α in training is RReLU.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
