Definition
SELU (scaled exponential linear unit) is for and for , with the fixed constants and . Combined with LeCun-normal initialization it makes a deep feed-forward network self-normalizing: activation means and variances converge to layer by layer, without batch normalization.
Batch normalization keeps activations at zero mean and unit variance by measuring and correcting them. Klambauer, Unterthiner, Mayr and Hochreiter asked whether the activation function could do the job instead ("Self-Normalizing Neural Networks", NeurIPS 2017). They chose the two free numbers in a scaled ELU so that a layer maps zero mean and unit variance to zero mean and unit variance, and proved that nearby moments are pulled toward that point.
With it they trained fully connected networks with many layers without normalization layers. Self-normalizing networks (SNNs) beat the other feed-forward variants they compared (BatchNorm, LayerNorm, WeightNorm, highway and residual MLPs) on 121 UCI classification tasks, and did best on the Tox21 drug-toxicity benchmark and the HTRU2 pulsar dataset. Their best models were often deep, where most other MLP variants worked best shallow.
Formula and derivative
- scale; , so the right branch can increase a variance that is too small
- ELU's negative saturation; the output floor is
- left slope: at against at , so SELU has a kink in its derivative
The constants are not tuned by experiment; they solve two equations. For require and . The mean condition fixes alone, because multiplies everything:
- standard normal CDF: ,
- , the positive half's second moment
- , the negative half's
Properties
- Range
- Monotonic
- yes, strictly increasing
- Smooth
- continuous; the slope jumps from to at 0
- Zero-centred
- by construction: mean exactly 0 and variance exactly 1 for input
- Saturating
- softly on the left (toward ), never on the right
- Requirements
- LeCun-normal init, AlphaDropout instead of dropout, plain dense layers, (0,1)-standardised inputs
- Cost
- ELU's: one exp per negative input
The moment map and its fixed point
Consider one unit whose inputs have mean and variance , with weights summarised by and . By the central limit theorem its pre-activation is approximately , and the output's moments are a deterministic function of the input's:
- the map g applied once per layer; a deep net iterates it
- normalized weights; LeCun normal gives this in expectation
- a fixed point of by the choice of
Theorem 1 of the paper shows that is a stable and attracting fixed point: is a contraction on a neighbourhood of it (the theorem's domain is , , ). Two further theorems bound the variance from above and below, so it can neither explode nor vanish even starting outside that region. The mechanism is visible in the formula: when is small, the inputs mostly use the right branch with slope , which raises the variance; when is large, more inputs land on the saturating left branch, which caps it.
Worked example: iterating the map
| k | SELU from (1, 2) | ReLU from (0, 1) | tanh from (0, 1) |
|---|---|---|---|
| 0 | (1, 2) | (0, 1) | (0, 1) |
| 1 | (0.090, 1.714) | (0.399, 0.341) | (0, 0.394) |
| 2 | (0.064, 1.521) | (0.233, 0.116) | (0, 0.236) |
| 5 | (0.026, 1.220) | (0.046, 0.005) | (0, 0.103) |
| 10 | (0.007, 1.060) | (0.003, 0.000) | (0, 0.052) |
| 64 | (0.000, 1.000) | (0.000, 0.000) | (0, 0.008) |
The first step does most of the mean correction: with the input mean cannot leak through the weights, so only SELU's own asymmetry sets . The variance then decays geometrically toward 1. ReLU and tanh have no such attractor at : their maps only shrink the variance (ReLU halves the second moment; tanh compresses it). ReLU needs , He init, to hold the variance; SELU needs to hold it at exactly 1.
What it requires
| requirement | why |
|---|---|
| LeCun-normal init, w ~ N(0, 1/fan_in) | gives , . He init has , which moves the attracting point and, in the figure, makes the variance explode |
| AlphaDropout, not dropout | standard dropout sets units to 0 and rescales the rest by , which changes the variance. AlphaDropout sets dropped units to SELU's floor and applies an affine correction; for that is with , |
| Standardised inputs | the argument assumes the first layer's inputs are near zero mean and unit variance |
| Plain dense stacks | a residual sum of two unit-variance streams has variance near 2, and BatchNorm or LayerNorm re-impose their own statistics, so the fixed-point argument no longer describes the network |
1import torch, torch.nn as nn, torch.nn.functional as F23LAMBDA, ALPHA = 1.0507009873554805, 1.673263242354377245def selu(x): # from scratch (autograd handles the backward)6 return LAMBDA * torch.where(x > 0, x, ALPHA * torch.expm1(x))78x = torch.tensor([-3.0, -1.0, -0.5, 0.5, 2.0])9print(F.selu(x)) # tensor([-1.6706, -1.1113, -0.6918, 0.5254, 2.1014])10print(selu(x)) # same values; nn.SELU() is the module form1112def snn(d_in, width=256, depth=16, p=0.05):13 layers = []14 for i in range(depth):15 lin = nn.Linear(d_in if i == 0 else width, width)16 # LeCun normal: std = 1/sqrt(fan_in). Use "linear", NOT "selu" (that gain is 3/4)17 nn.init.kaiming_normal_(lin.weight, nonlinearity="linear")18 nn.init.zeros_(lin.bias)19 layers += [lin, nn.SELU(), nn.AlphaDropout(p)]20 return nn.Sequential(*layers, nn.Linear(width, 1))2122net = snn(32).eval() # eval(): AlphaDropout is the identity23with torch.no_grad():24 h = torch.randn(4096, 32) # standardised inputs25 for m in net:26 h = m(h)27 if isinstance(m, nn.SELU):28 print(f"mean {h.mean():+.3f} var {h.var():.3f}") # stays close to (0, 1)Use when
- +Normalization without normalization layers: no batch statistics, works at batch size 1
- +Deep MLPs train stably; proven attracting fixed point at (0,1)
- +No dead units; mean zero by construction
- +Same cost as ELU, available everywhere (nn.SELU, AlphaDropout)
Avoid when
- −Only valid with LeCun-normal init, AlphaDropout and standardised inputs
- −Theory covers plain dense stacks, not residual or attention architectures
- −Fixed point shifts with weight scale τ, and weights drift during training
- −Rarely used in CNNs and transformers, where BatchNorm/LayerNorm dominate
Check yourself
A SELU network is initialised with He-normal weights, Var(w) = 2/fan_in, instead of LeCun normal. What happens to the activation variance through depth?
Check yourself
Why does SELU need AlphaDropout rather than ordinary dropout?
Remember
- ✓ with , : the unique pair giving mean 0 and variance 1 for input.
- ✓The moment map has a stable, attracting fixed point at for , (Klambauer et al., 2017).
- ✓Requires LeCun-normal init (), AlphaDropout and standardised inputs; He init or the "selu" gain break it.
- ✓Works for plain deep MLPs; BatchNorm, residual sums and attention architectures fall outside the theory.
- ✓ lets small variances grow and the saturating left branch caps large ones: both are needed for an attractor.