Parametric ReLU (PReLU)

Leaky ReLU with the negative slope a learned by backprop, one per channel: ∂L/∂a sums z·δ over the negative inputs. He et al. (2015) introduced it with the init Var(w) = 2/((1+a²)n).

9 min read666 wordsadvanced0 viewsupdated 2026-10-11

Definition

Parametric ReLU (PReLU) is the activation f(x)=max⁡(0,x)+amin⁡(0,x)f(x)=\max(0,x)+a\min(0,x), where the negative slope aa is a trainable parameter, usually one per channel, updated by gradient descent together with the weights.

Leaky ReLU fixes the negative slope by hand. PReLU asks the data instead: if a layer works better with more of the negative signal passing through, the gradient on aa will say so. The extra cost is one scalar per channel, negligible next to the weights.

He, Zhang, Ren and Sun introduced it in "Delving Deep into Rectifiers" (ICCV 2015), the same paper that derived He (Kaiming) initialization. Their PReLU network reached 4.94% top-5 error on ImageNet 2012, the first published result below the 5.1% estimate of human error on that benchmark. The learned slopes were far from zero, and in the channel-wise model the deeper conv layers generally learned smaller ones than the early layers.

Formula and derivatives

f(xi)={xixi>0ac xixi≤0∂f∂xi={1xi>0acxi<0∂f∂ac={0xi>0xixi≤0f(x_i)=\begin{cases}x_i & x_i>0\\ a_c\,x_i & x_i\le 0\end{cases}\qquad\frac{\partial f}{\partial x_i}=\begin{cases}1 & x_i>0\\ a_c & x_i<0\end{cases}\qquad\frac{\partial f}{\partial a_c}=\begin{cases}0 & x_i>0\\ x_i & x_i\le0\end{cases}
aca_c
the slope of channel cc, initialised to 0.25 in the paper and in PyTorch
∂f/∂xi\partial f/\partial x_i
the input derivative, exactly as in Leaky ReLU
∂f/∂ac\partial f/\partial a_c
the new one: the input itself on the negative side, 0 on the positive side

A slope shared by many positions collects gradient from all of them. With δi=∂L/∂f(xi)\delta_i=\partial\mathcal L/\partial f(x_i) the error arriving at each output, the chain rule sums over every position that uses aca_c, across the batch and the spatial grid:

∂L∂ac=∑i ∈ channel cδi ∂f∂ac=∑i∈c,  xi<0xi δi\frac{\partial\mathcal L}{\partial a_c}=\sum_{i\,\in\,\text{channel } c}\delta_i\,\frac{\partial f}{\partial a_c}=\sum_{i\in c,\;x_i<0}x_i\,\delta_i
xi<0x_i<0
only negative inputs contribute; positive ones see no dependence on a
xiδix_i\delta_i
negative when the loss wants the output more negative (δi>0\delta_i>0), so gradient descent raises aca_c there

Channel-wise PReLU learns one aca_c per channel (nn.PReLU(num_parameters=C)); channel-shared learns one aa per layer (nn.PReLU()). He et al. tested both; they performed similarly and both beat ReLU. a=0a=0 recovers ReLU, a small fixed aa is Leaky ReLU, and a=1a=1 makes the unit linear. Nothing constrains aa; it can even go negative, which makes the unit non-monotonic.

Properties

Range
(−∞,∞)(-\infty,\infty) for a>0a>0; [0,∞)[0,\infty) at a=0a=0
Monotonic
if a≥0a\ge0; learned a<0a<0 gives a V-shaped, non-monotonic unit
Smooth
piecewise linear, kink at 0 unless a=1a=1
Zero-centred
no; for N(0,1)\mathcal N(0,1) input and a=0.25a=0.25 the mean output is 0.2990.299
Saturating
no: slope 1 on the right and aa on the left
Parameters
one per channel (or one per layer); He et al. counted them as negligible
Cost
Leaky ReLU's cost, plus one reduction per channel for ∂L/∂a
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 1.000 · f′(x) = 1.000−2024−4−2024x →PReLU (a=0.25)PReLU f′ReLULeaky ReLU
activation
compare
range (−∞, ∞)monotonic yessmooth nozero-centred no
Leaky ReLU with a learned slope a (PyTorch initialises it to 0.25). f′(0) is taken as a; monotonic only for a ≥ 0.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]−1.001.504.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.299Var[f(x)] 0.442E[f′(x)] 0.625f′ = 0 for 0.0% of inputsE[f′]¹⁰ 9.1e−3
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.PReLU at its initial slope a = 0.25, against ReLU and Leaky ReLU (α = 0.01). Drag a: this is the one knob that training turns. At a = 0 the curve is ReLU, at a = 1 it is the identity.

Initialization for PReLU

He et al. derived their initialization for this very activation. For a symmetric zero-mean pre-activation, PReLU keeps a fraction 1+a22\tfrac{1+a^2}{2} of the second moment: half of the mass passes unchanged and the other half is scaled by a2a^2. Keeping the variance constant from layer to layer needs:

12(1+a2) nlVar⁡(wl)=1⟹Var⁡(wl)=2(1+a2) nl\tfrac12(1+a^2)\,n_l\operatorname{Var}(w_l)=1\quad\Longrightarrow\quad \operatorname{Var}(w_l)=\frac{2}{(1+a^2)\,n_l}
nln_l
fan-in of layer ll (k2ck^2c for a k×kk\times k convolution with cc input channels)
a=0a=0
gives the familiar ReLU rule Var⁡(w)=2/nl\operatorname{Var}(w)=2/n_l
a=0.25a=0.25
gain 2/1.0625=1.3720\sqrt{2/1.0625}=1.3720 instead of ReLU's 1.41421.4142

The backward pass gives the same condition with the fan-out in place of the fan-in. He et al. used it to train a 30-layer plain network from scratch, where Xavier initialization stalled. For ReLU-family nets the difference between the two rules is the factor 1+a22\tfrac{1+a^2}{2}, which compounds over depth.

Learning the slope

31 points, fit ŷ = PReLU_a(wx + b)−2−10123−3−2−10123z < 0 side: 15 points feed ∂L/∂aL(a), w and b held at current values11e-11e-21e-300.51aa over GD steps−0.500.51a*0100200300400step 0: a = 0.000, w = 0.800, b = 0.000, L = 0.3144∂L/∂a = (2/N)·Σ over z<0 of z·(ŷ−y) = −0.8815
GD on (w, b, a), lr 0.08
fig.One PReLU unit ŷ = PReLU_a(wx + b) fit by gradient descent (w, b and a together) to 31 points generated with a true slope a* = 0.25 plus noise. Top right: the loss as a function of a with w and b held at their current values; orange dots are earlier steps. Starting from a = 0 (ReLU) the gradient is ∂L/∂a = −0.88, so a rises fast, overshoots to 0.319 by step 25 while w and b catch up, and settles at a = 0.268, MSE 0.0039. Any starting a reaches the same value. Turn on weight decay: λ = 0.1 pulls the final a to 0.249 and λ = 1 to 0.178, biasing the unit toward ReLU.

The slope is learned like any weight, but with two conventions from the paper: the same learning rate and momentum as the weights, and no weight decay on aa. Decay adds 2λa2\lambda a to the gradient and pulls every slope toward 0, that is, toward ReLU, whatever the data says. PyTorch's documentation gives the same advice.

Worked example: one backward step for a

A channel with a=0.25a=0.25 sees five pre-activations x=(−2, −0.5, 0.3, 1.5, −1)x=(-2,\,-0.5,\,0.3,\,1.5,\,-1), and the layer above sends back δ=(0.4, −0.2, 0.1, 0.3, 0.5)\delta=(0.4,\,-0.2,\,0.1,\,0.3,\,0.5).

xf(x)δ = ∂L/∂f∂L/∂x = δ·f′(x)contribution x·δ to ∂L/∂a
−2−0.50.40.1−0.8
−0.5−0.125−0.2−0.050.1
0.30.30.10.1— (positive)
1.51.50.30.3— (positive)
−1−0.250.50.125−0.5
sum: −1.2
Forward and backward through PReLU with a = 0.25 (computed in Node)

So ∂L/∂a=−1.2\partial\mathcal L/\partial a=-1.2, and an SGD step with learning rate 0.010.01 raises the slope from 0.250.25 to 0.2620.262. The first and last inputs dominate: they are negative and their outputs should be more negative (δ>0\delta>0 means decreasing the output lowers the loss), which a steeper slope delivers.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-2.0, -0.5, 0.3, 1.5, -1.0], requires_grad=True)
4delta = torch.tensor([0.4, -0.2, 0.1, 0.3, 0.5])
5
6act = nn.PReLU(num_parameters=1, init=0.25) # one shared a; use num_parameters=C per channel
7y = act(x) # same as F.prelu(x, act.weight)
8y.backward(delta)
9print(act.weight.grad) # tensor([-1.2000]) = sum of x * delta over x < 0
10print(x.grad) # tensor([ 0.1000, -0.0500, 0.1000, 0.3000, 0.1250])
11
12def prelu(x, a): # from scratch; for (N, C, H, W) inputs pass a of shape (1, C, 1, 1)
13 return torch.where(x > 0, x, a * x)
14
15# Exclude the slopes (and usually biases/norm params) from weight decay
16model = nn.Sequential(nn.Conv2d(3, 64, 3, padding=1), nn.PReLU(64), nn.Conv2d(64, 64, 3, padding=1), nn.PReLU(64))
17slopes = [m.weight for m in model.modules() if isinstance(m, nn.PReLU)]
18others = [p for p in model.parameters() if all(p is not s for s in slopes)]
19opt = torch.optim.SGD([{"params": others, "weight_decay": 1e-4},
20 {"params": slopes, "weight_decay": 0.0}], lr=0.1, momentum=0.9)
PReLU in PyTorch: module, functional, from scratch, and an optimizer without weight decay on a

Use when

  • +The data picks the negative slope; no α to tune
  • +No dead units once a > 0, like Leaky ReLU
  • +Negligible parameter and compute overhead
  • +Came with a matching init: Var(w) = 2/((1+a²)n)

Avoid when

  • −Needs weight decay disabled on a, an easy thing to forget
  • −Can overfit small datasets
  • −Learned a can drift to 1 (linear) or below 0 (non-monotonic)
  • −Inherits the kink and the non-zero mean

Check yourself

In a channel-wise PReLU, which inputs contribute to ∂L/∂a_c?

Check yourself

Why did He et al. not apply weight decay to the PReLU slopes?

Remember

  • ✓f(x)=max⁡(0,x)+amin⁡(0,x)f(x)=\max(0,x)+a\min(0,x) with aa learned, one per channel or one per layer; initialised at 0.25.
  • ✓∂f/∂a=x\partial f/\partial a=x on the negative side, so ∂L/∂ac=∑xi<0xiδi\partial\mathcal L/\partial a_c=\sum_{x_i<0}x_i\delta_i over all positions of channel cc.
  • ✓He et al. (2015) derived Var⁡(w)=2/((1+a2)nl)\operatorname{Var}(w)=2/((1+a^2)n_l) for it, and reached 4.94% top-5 error on ImageNet.
  • ✓Do not apply weight decay to a: it biases the unit toward ReLU.
  • ✓a = 0 is ReLU, small fixed a is Leaky ReLU, a = 1 is linear.
← back to Deep Learning roadmap