# Parametric ReLU (PReLU)

> Leaky ReLU with the negative slope a learned by backprop, one per channel: ∂L/∂a sums z·δ over the negative inputs. He et al. (2015) introduced it with the init Var(w) = 2/((1+a²)n).

Source: https://www.mlfoundry.in/learn/parametric-relu-prelu  
Section: Deep Learning › Activation Functions  
Level: advanced · 9 min read · updated 2026-10-11

**Parametric ReLU:** (PReLU) is the activation $f(x)=\max(0,x)+a\min(0,x)$, where the negative slope $a$ is a **trainable parameter**, usually one per channel, updated by gradient descent together with the weights.

Leaky ReLU fixes the negative slope by hand. PReLU asks the data instead: if a layer works better with more of the negative signal passing through, the gradient on $a$ will say so. The extra cost is one scalar per channel, negligible next to the weights.

He, Zhang, Ren and Sun introduced it in "Delving Deep into Rectifiers" (ICCV 2015), the same paper that derived He (Kaiming) initialization. Their PReLU network reached 4.94% top-5 error on ImageNet 2012, the first published result below the 5.1% estimate of human error on that benchmark. The learned slopes were far from zero, and in the channel-wise model the deeper conv layers generally learned smaller ones than the early layers.

## Formula and derivatives

$$
f(x_i)=\begin{cases}x_i & x_i>0\\ a_c\,x_i & x_i\le 0\end{cases}\qquad\frac{\partial f}{\partial x_i}=\begin{cases}1 & x_i>0\\ a_c & x_i<0\end{cases}\qquad\frac{\partial f}{\partial a_c}=\begin{cases}0 & x_i>0\\ x_i & x_i\le0\end{cases}
$$

- $a_c$: the slope of channel $c$, initialised to 0.25 in the paper and in PyTorch
- $\partial f/\partial x_i$: the input derivative, exactly as in Leaky ReLU
- $\partial f/\partial a_c$: the new one: the input itself on the negative side, 0 on the positive side

A slope shared by many positions collects gradient from all of them. With $\delta_i=\partial\mathcal L/\partial f(x_i)$ the error arriving at each output, the chain rule sums over every position that uses $a_c$, across the batch and the spatial grid:

$$
\frac{\partial\mathcal L}{\partial a_c}=\sum_{i\,\in\,\text{channel } c}\delta_i\,\frac{\partial f}{\partial a_c}=\sum_{i\in c,\;x_i<0}x_i\,\delta_i
$$

- $x_i<0$: only negative inputs contribute; positive ones see no dependence on a
- $x_i\delta_i$: negative when the loss wants the output more negative ($\delta_i>0$), so gradient descent raises $a_c$ there

**Channel-wise** PReLU learns one $a_c$ per channel (`nn.PReLU(num_parameters=C)`); **channel-shared** learns one $a$ per layer (`nn.PReLU()`). He et al. tested both; they performed similarly and both beat ReLU. $a=0$ recovers ReLU, a small fixed $a$ is Leaky ReLU, and $a=1$ makes the unit linear. Nothing constrains $a$; it can even go negative, which makes the unit non-monotonic.

**Properties**

- **Range:** $(-\infty,\infty)$ for $a>0$; $[0,\infty)$ at $a=0$
- **Monotonic:** if $a\ge0$; learned $a<0$ gives a V-shaped, non-monotonic unit
- **Smooth:** piecewise linear, kink at 0 unless $a=1$
- **Zero-centred:** no; for $\mathcal N(0,1)$ input and $a=0.25$ the mean output is $0.299$
- **Saturating:** no: slope 1 on the right and $a$ on the left
- **Parameters:** one per channel (or one per layer); He et al. counted them as negligible
- **Cost:** Leaky ReLU's cost, plus one reduction per channel for ∂L/∂a

> Figure: PReLU at its initial slope a = 0.25, against ReLU and Leaky ReLU (α = 0.01). Drag a: this is the one knob that training turns. At a = 0 the curve is ReLU, at a = 1 it is the identity.

## Initialization for PReLU

He et al. derived their initialization for this very activation. For a symmetric zero-mean pre-activation, PReLU keeps a fraction $\tfrac{1+a^2}{2}$ of the second moment: half of the mass passes unchanged and the other half is scaled by $a^2$. Keeping the variance constant from layer to layer needs:

$$
\tfrac12(1+a^2)\,n_l\operatorname{Var}(w_l)=1\quad\Longrightarrow\quad \operatorname{Var}(w_l)=\frac{2}{(1+a^2)\,n_l}
$$

- $n_l$: fan-in of layer $l$ ($k^2c$ for a $k\times k$ convolution with $c$ input channels)
- $a=0$: gives the familiar ReLU rule $\operatorname{Var}(w)=2/n_l$
- $a=0.25$: gain $\sqrt{2/1.0625}=1.3720$ instead of ReLU's $1.4142$

The backward pass gives the same condition with the fan-out in place of the fan-in. He et al. used it to train a 30-layer plain network from scratch, where Xavier initialization stalled. For ReLU-family nets the difference between the two rules is the factor $\tfrac{1+a^2}{2}$, which compounds over depth.

## Learning the slope

> Figure: One PReLU unit ŷ = PReLU_a(wx + b) fit by gradient descent (w, b and a together) to 31 points generated with a true slope a* = 0.25 plus noise. Top right: the loss as a function of a with w and b held at their current values; orange dots are earlier steps. Starting from a = 0 (ReLU) the gradient is ∂L/∂a = −0.88, so a rises fast, overshoots to 0.319 by step 25 while w and b catch up, and settles at a = 0.268, MSE 0.0039. Any starting a reaches the same value. Turn on weight decay: λ = 0.1 pulls the final a to 0.249 and λ = 1 to 0.178, biasing the unit toward ReLU.

The slope is learned like any weight, but with two conventions from the paper: the same learning rate and momentum as the weights, and **no weight decay** on $a$. Decay adds $2\lambda a$ to the gradient and pulls every slope toward 0, that is, toward ReLU, whatever the data says. PyTorch's documentation gives the same advice.

## Worked example: one backward step for a

A channel with $a=0.25$ sees five pre-activations $x=(-2,\,-0.5,\,0.3,\,1.5,\,-1)$, and the layer above sends back $\delta=(0.4,\,-0.2,\,0.1,\,0.3,\,0.5)$.

Forward and backward through PReLU with a = 0.25 (computed in Node)

| x | f(x) | δ = ∂L/∂f | ∂L/∂x = δ·f′(x) | contribution x·δ to ∂L/∂a |
| --- | --- | --- | --- | --- |
| −2 | −0.5 | 0.4 | 0.1 | −0.8 |
| −0.5 | −0.125 | −0.2 | −0.05 | 0.1 |
| 0.3 | 0.3 | 0.1 | 0.1 | — (positive) |
| 1.5 | 1.5 | 0.3 | 0.3 | — (positive) |
| −1 | −0.25 | 0.5 | 0.125 | −0.5 |
|  |  |  |  | **sum: −1.2** |

So $\partial\mathcal L/\partial a=-1.2$, and an SGD step with learning rate $0.01$ raises the slope from $0.25$ to $0.262$. The first and last inputs dominate: they are negative and their outputs should be more negative ($\delta>0$ means decreasing the output lowers the loss), which a steeper slope delivers.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-2.0, -0.5, 0.3, 1.5, -1.0], requires_grad=True)
delta = torch.tensor([0.4, -0.2, 0.1, 0.3, 0.5])

act = nn.PReLU(num_parameters=1, init=0.25)   # one shared a; use num_parameters=C per channel
y = act(x)                                    # same as F.prelu(x, act.weight)
y.backward(delta)
print(act.weight.grad)   # tensor([-1.2000])  = sum of x * delta over x < 0
print(x.grad)            # tensor([ 0.1000, -0.0500,  0.1000,  0.3000,  0.1250])

def prelu(x, a):         # from scratch; for (N, C, H, W) inputs pass a of shape (1, C, 1, 1)
    return torch.where(x > 0, x, a * x)

# Exclude the slopes (and usually biases/norm params) from weight decay
model = nn.Sequential(nn.Conv2d(3, 64, 3, padding=1), nn.PReLU(64), nn.Conv2d(64, 64, 3, padding=1), nn.PReLU(64))
slopes = [m.weight for m in model.modules() if isinstance(m, nn.PReLU)]
others = [p for p in model.parameters() if all(p is not s for s in slopes)]
opt = torch.optim.SGD([{"params": others, "weight_decay": 1e-4},
                       {"params": slopes, "weight_decay": 0.0}], lr=0.1, momentum=0.9)
```

PReLU in PyTorch: module, functional, from scratch, and an optimizer without weight decay on a

> **Weight decay on a:** Optimizers apply weight decay to every parameter in their list unless told otherwise, so a model with `weight_decay=1e-4` quietly regularises its PReLU slopes toward ReLU. In the figure, $\lambda=1$ moves the learned slope from $0.268$ to $0.178$. Put the slopes in a parameter group with `weight_decay=0`.

> **What a learned slope tells you:** A learned $a$ is a cheap diagnostic. Slopes near 0 mean the layer prefers ReLU-like sparse gating; slopes far from 0 mean it wants to keep the negative signal. The same idea, letting the data choose the shape of the nonlinearity, reappears in Swish with a learnable $\beta$ and in learned spline activations.

> **Overfitting risk:** He et al. described the extra overfitting risk on ImageNet as small. On small datasets the extra freedom matters more: Xu et al. (2015) reported that PReLU overfits easily on CIFAR-sized data, and that the randomized RReLU generalised better there.

**Strengths**

- The data picks the negative slope; no α to tune
- No dead units once a > 0, like Leaky ReLU
- Negligible parameter and compute overhead
- Came with a matching init: Var(w) = 2/((1+a²)n)

**Limitations**

- Needs weight decay disabled on a, an easy thing to forget
- Can overfit small datasets
- Learned a can drift to 1 (linear) or below 0 (non-monotonic)
- Inherits the kink and the non-zero mean

**Question:** In a channel-wise PReLU, which inputs contribute to ∂L/∂a_c?

- A. Every input in the batch, positive and negative
- B. Only negative inputs in channel c, each weighted by its upstream error
- C. Only positive inputs in channel c
- D. Only the input with the most negative value

**Answer:** B. Only negative inputs in channel c, each weighted by its upstream error — $\partial f/\partial a_c=x_i$ for $x_i\le0$ and 0 for $x_i>0$, so $\partial\mathcal L/\partial a_c=\sum_{i\in c,\,x_i<0}x_i\delta_i$, summed over the batch and every spatial position in that channel.

**Question:** Why did He et al. not apply weight decay to the PReLU slopes?

- A. It makes the slopes explode
- B. It pulls every a toward 0, biasing PReLU toward ReLU regardless of the data
- C. Weight decay cannot be applied to scalar parameters
- D. It would make the init Var(w) = 2/((1+a²)n) invalid

**Answer:** B. It pulls every a toward 0, biasing PReLU toward ReLU regardless of the data — L2 regularisation adds $2\lambda a$ to $\partial\mathcal L/\partial a$, a constant pull toward $a=0$. The slopes are few and cheap, so there is no overfitting argument for shrinking them, only a bias toward ReLU.

**Key points**

- $f(x)=\max(0,x)+a\min(0,x)$ with $a$ learned, one per channel or one per layer; initialised at 0.25.
- $\partial f/\partial a=x$ on the negative side, so $\partial\mathcal L/\partial a_c=\sum_{x_i<0}x_i\delta_i$ over all positions of channel $c$.
- He et al. (2015) derived $\operatorname{Var}(w)=2/((1+a^2)n_l)$ for it, and reached 4.94% top-5 error on ImageNet.
- Do not apply weight decay to a: it biases the unit toward ReLU.
- a = 0 is ReLU, small fixed a is Leaky ReLU, a = 1 is linear.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
