Definition
Parametric ReLU (PReLU) is the activation , where the negative slope is a trainable parameter, usually one per channel, updated by gradient descent together with the weights.
Leaky ReLU fixes the negative slope by hand. PReLU asks the data instead: if a layer works better with more of the negative signal passing through, the gradient on will say so. The extra cost is one scalar per channel, negligible next to the weights.
He, Zhang, Ren and Sun introduced it in "Delving Deep into Rectifiers" (ICCV 2015), the same paper that derived He (Kaiming) initialization. Their PReLU network reached 4.94% top-5 error on ImageNet 2012, the first published result below the 5.1% estimate of human error on that benchmark. The learned slopes were far from zero, and in the channel-wise model the deeper conv layers generally learned smaller ones than the early layers.
Formula and derivatives
- the slope of channel , initialised to 0.25 in the paper and in PyTorch
- the input derivative, exactly as in Leaky ReLU
- the new one: the input itself on the negative side, 0 on the positive side
A slope shared by many positions collects gradient from all of them. With the error arriving at each output, the chain rule sums over every position that uses , across the batch and the spatial grid:
- only negative inputs contribute; positive ones see no dependence on a
- negative when the loss wants the output more negative (), so gradient descent raises there
Channel-wise PReLU learns one per channel (nn.PReLU(num_parameters=C)); channel-shared learns one per layer (nn.PReLU()). He et al. tested both; they performed similarly and both beat ReLU. recovers ReLU, a small fixed is Leaky ReLU, and makes the unit linear. Nothing constrains ; it can even go negative, which makes the unit non-monotonic.
Properties
- Range
- for ; at
- Monotonic
- if ; learned gives a V-shaped, non-monotonic unit
- Smooth
- piecewise linear, kink at 0 unless
- Zero-centred
- no; for input and the mean output is
- Saturating
- no: slope 1 on the right and on the left
- Parameters
- one per channel (or one per layer); He et al. counted them as negligible
- Cost
- Leaky ReLU's cost, plus one reduction per channel for ∂L/∂a
Initialization for PReLU
He et al. derived their initialization for this very activation. For a symmetric zero-mean pre-activation, PReLU keeps a fraction of the second moment: half of the mass passes unchanged and the other half is scaled by . Keeping the variance constant from layer to layer needs:
- fan-in of layer ( for a convolution with input channels)
- gives the familiar ReLU rule
- gain instead of ReLU's
The backward pass gives the same condition with the fan-out in place of the fan-in. He et al. used it to train a 30-layer plain network from scratch, where Xavier initialization stalled. For ReLU-family nets the difference between the two rules is the factor , which compounds over depth.
Learning the slope
The slope is learned like any weight, but with two conventions from the paper: the same learning rate and momentum as the weights, and no weight decay on . Decay adds to the gradient and pulls every slope toward 0, that is, toward ReLU, whatever the data says. PyTorch's documentation gives the same advice.
Worked example: one backward step for a
A channel with sees five pre-activations , and the layer above sends back .
| x | f(x) | δ = ∂L/∂f | ∂L/∂x = δ·f′(x) | contribution x·δ to ∂L/∂a |
|---|---|---|---|---|
| −2 | −0.5 | 0.4 | 0.1 | −0.8 |
| −0.5 | −0.125 | −0.2 | −0.05 | 0.1 |
| 0.3 | 0.3 | 0.1 | 0.1 | — (positive) |
| 1.5 | 1.5 | 0.3 | 0.3 | — (positive) |
| −1 | −0.25 | 0.5 | 0.125 | −0.5 |
| sum: −1.2 |
So , and an SGD step with learning rate raises the slope from to . The first and last inputs dominate: they are negative and their outputs should be more negative ( means decreasing the output lowers the loss), which a steeper slope delivers.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([-2.0, -0.5, 0.3, 1.5, -1.0], requires_grad=True)4delta = torch.tensor([0.4, -0.2, 0.1, 0.3, 0.5])56act = nn.PReLU(num_parameters=1, init=0.25) # one shared a; use num_parameters=C per channel7y = act(x) # same as F.prelu(x, act.weight)8y.backward(delta)9print(act.weight.grad) # tensor([-1.2000]) = sum of x * delta over x < 010print(x.grad) # tensor([ 0.1000, -0.0500, 0.1000, 0.3000, 0.1250])1112def prelu(x, a): # from scratch; for (N, C, H, W) inputs pass a of shape (1, C, 1, 1)13 return torch.where(x > 0, x, a * x)1415# Exclude the slopes (and usually biases/norm params) from weight decay16model = nn.Sequential(nn.Conv2d(3, 64, 3, padding=1), nn.PReLU(64), nn.Conv2d(64, 64, 3, padding=1), nn.PReLU(64))17slopes = [m.weight for m in model.modules() if isinstance(m, nn.PReLU)]18others = [p for p in model.parameters() if all(p is not s for s in slopes)]19opt = torch.optim.SGD([{"params": others, "weight_decay": 1e-4},20 {"params": slopes, "weight_decay": 0.0}], lr=0.1, momentum=0.9)Use when
- +The data picks the negative slope; no α to tune
- +No dead units once a > 0, like Leaky ReLU
- +Negligible parameter and compute overhead
- +Came with a matching init: Var(w) = 2/((1+a²)n)
Avoid when
- −Needs weight decay disabled on a, an easy thing to forget
- −Can overfit small datasets
- −Learned a can drift to 1 (linear) or below 0 (non-monotonic)
- −Inherits the kink and the non-zero mean
Check yourself
In a channel-wise PReLU, which inputs contribute to ∂L/∂a_c?
Check yourself
Why did He et al. not apply weight decay to the PReLU slopes?
Remember
- ✓ with learned, one per channel or one per layer; initialised at 0.25.
- ✓ on the negative side, so over all positions of channel .
- ✓He et al. (2015) derived for it, and reached 4.94% top-5 error on ImageNet.
- ✓Do not apply weight decay to a: it biases the unit toward ReLU.
- ✓a = 0 is ReLU, small fixed a is Leaky ReLU, a = 1 is linear.