SELU

λ·ELU_α with λ ≈ 1.0507 and α ≈ 1.6733, chosen so that zero mean and unit variance are an attracting fixed point of every dense layer: with LeCun-normal weights the net normalises itself.

10 min read640 wordsadvanced0 viewsupdated 2026-10-11

Definition

SELU (scaled exponential linear unit) is SELU(x)=λx\mathrm{SELU}(x)=\lambda x for x>0x>0 and λα(ex−1)\lambda\alpha(e^{x}-1) for x≤0x\le0, with the fixed constants λ≈1.0507\lambda\approx1.0507 and α≈1.6733\alpha\approx1.6733. Combined with LeCun-normal initialization it makes a deep feed-forward network self-normalizing: activation means and variances converge to (0,1)(0,1) layer by layer, without batch normalization.

Batch normalization keeps activations at zero mean and unit variance by measuring and correcting them. Klambauer, Unterthiner, Mayr and Hochreiter asked whether the activation function could do the job instead ("Self-Normalizing Neural Networks", NeurIPS 2017). They chose the two free numbers in a scaled ELU so that a layer maps zero mean and unit variance to zero mean and unit variance, and proved that nearby moments are pulled toward that point.

With it they trained fully connected networks with many layers without normalization layers. Self-normalizing networks (SNNs) beat the other feed-forward variants they compared (BatchNorm, LayerNorm, WeightNorm, highway and residual MLPs) on 121 UCI classification tasks, and did best on the Tox21 drug-toxicity benchmark and the HTRU2 pulsar dataset. Their best models were often deep, where most other MLP variants worked best shallow.

Formula and derivative

SELU(x)=λ{xx>0α (ex−1)x≤0SELU′(x)={λx>0λα ex=SELU(x)+λαx≤0\mathrm{SELU}(x)=\lambda\begin{cases}x & x>0\\ \alpha\,(e^{x}-1) & x\le 0\end{cases}\qquad\qquad \mathrm{SELU}'(x)=\begin{cases}\lambda & x>0\\ \lambda\alpha\,e^{x}=\mathrm{SELU}(x)+\lambda\alpha & x\le 0\end{cases}
λ=1.0507009873554805\lambda=1.0507009873554805
scale; >1>1, so the right branch can increase a variance that is too small
α=1.6732632423543772\alpha=1.6732632423543772
ELU's negative saturation; the output floor is −λα=−1.7581-\lambda\alpha=-1.7581
λαex\lambda\alpha e^{x}
left slope: 1.75811.7581 at 0−0^- against 1.05071.0507 at 0+0^+, so SELU has a kink in its derivative

The constants are not tuned by experiment; they solve two equations. For z∼N(0,1)z\sim\mathcal N(0,1) require E[SELU(z)]=0\mathbb E[\mathrm{SELU}(z)]=0 and E[SELU(z)2]=1\mathbb E[\mathrm{SELU}(z)^2]=1. The mean condition fixes α\alpha alone, because λ\lambda multiplies everything:

α=1/2π12−e1/2Φ(−1)=1.6733,λ=(12+α2(e2Φ(−2)−2e1/2Φ(−1)+12))−1/2=1.0507\alpha=\frac{1/\sqrt{2\pi}}{\tfrac12-e^{1/2}\Phi(-1)}=1.6733,\qquad \lambda=\Big(\tfrac12+\alpha^2\big(e^{2}\Phi(-2)-2e^{1/2}\Phi(-1)+\tfrac12\big)\Big)^{-1/2}=1.0507
Φ\Phi
standard normal CDF: e1/2Φ(−1)=0.26158e^{1/2}\Phi(-1)=0.26158, e2Φ(−2)=0.16810e^{2}\Phi(-2)=0.16810
12\tfrac12
E[z21[z>0]]\mathbb E[z^2\mathbb 1[z>0]], the positive half's second moment
α2(⋯ )\alpha^2(\cdots)
α2 E[(ez−1)21[z<0]]=α2⋅0.14495\alpha^2\,\mathbb E[(e^z-1)^2\mathbb 1[z<0]]=\alpha^2\cdot0.14495, the negative half's

Properties

Range
(−λα,∞)=(−1.7581,∞)(-\lambda\alpha,\infty)=(-1.7581,\infty)
Monotonic
yes, strictly increasing
Smooth
continuous; the slope jumps from 1.75811.7581 to 1.05071.0507 at 0
Zero-centred
by construction: mean exactly 0 and variance exactly 1 for N(0,1)\mathcal N(0,1) input
Saturating
softly on the left (toward −1.7581-1.7581), never on the right
Requirements
LeCun-normal init, AlphaDropout instead of dropout, plain dense layers, (0,1)-standardised inputs
Cost
ELU's: one exp per negative input
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 1.051 · f′(x) = 1.051−2024−4−2024x →SELUSELU f′ELUReLU
activation
compare
range (−λα, ∞) ≈ (−1.758, ∞)monotonic yessmooth nozero-centred yes
Fixed λ ≈ 1.0507, α ≈ 1.6733 chosen so N(0,1) inputs give zero-mean, unit-variance outputs. Kink at 0 (slopes λα ≈ 1.758 vs λ ≈ 1.051).
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]−1.731.244.20output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.000Var[f(x)] 1.000E[f′(x)] 0.985f′ = 0 for 0.0% of inputsE[f′]¹⁰ 0.862
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.SELU against ELU (α = 1) and ReLU. Note the floor at −1.758, the slope of 1.0507 on the right and the jump in the derivative at 0. For standard normal inputs the statistics panel shows SELU's output mean 0 and variance 1.0000, the property everything else is built on.

The moment map and its fixed point

Consider one unit whose nn inputs have mean μ\mu and variance ν\nu, with weights summarised by ω=∑iwi\omega=\sum_i w_i and τ=∑iwi2\tau=\sum_i w_i^2. By the central limit theorem its pre-activation is approximately N(μω,ντ)\mathcal N(\mu\omega,\nu\tau), and the output's moments are a deterministic function of the input's:

μ~=E[SELU(z)],ν~=E[SELU(z)2]−μ~2,z∼N(μω,  ντ)\tilde\mu=\mathbb E\big[\mathrm{SELU}(z)\big],\qquad \tilde\nu=\mathbb E\big[\mathrm{SELU}(z)^2\big]-\tilde\mu^2,\qquad z\sim\mathcal N(\mu\omega,\;\nu\tau)
(μ,ν)↦(μ~,ν~)(\mu,\nu)\mapsto(\tilde\mu,\tilde\nu)
the map g applied once per layer; a deep net iterates it
ω=0, τ=1\omega=0,\ \tau=1
normalized weights; LeCun normal w∼N(0,1/n)w\sim\mathcal N(0,1/n) gives this in expectation
(0,1)(0,1)
a fixed point of gg by the choice of λ,α\lambda,\alpha

Theorem 1 of the paper shows that (0,1)(0,1) is a stable and attracting fixed point: gg is a contraction on a neighbourhood of it (the theorem's domain is μ,ω∈[−0.1,0.1]\mu,\omega\in[-0.1,0.1], ν∈[0.8,1.5]\nu\in[0.8,1.5], τ∈[0.95,1.1]\tau\in[0.95,1.1]). Two further theorems bound the variance from above and below, so it can neither explode nor vanish even starting outside that region. The mechanism is visible in the formula: when ν\nu is small, the inputs mostly use the right branch with slope λ>1\lambda>1, which raises the variance; when ν\nu is large, more inputs land on the saturating left branch, which caps it.

SELU in the (mean μ, variance ν) plane0123−0.50.00.51.01.5μν● input ● moment map, ω=0, τ=s ● real layersvariance per layer (log)1e-41e-211e2floor: values below 1e-4 drawn at 1e-4mean per layer−0.50.01.02.0016324864layer━ selu layer 64: mean 0.057, var 0.968━ relu layer 64: mean 0.000, var 1.3e−20━ tanh layer 64: mean −0.006, var 0.008┅ map after 64 steps:(μ, ν) = (0.000, 1.000)
W ~ N(0, s/48), 64 layers, no biases
fig.Left: SELU's moment map iterated from the input (μ₀, ν₀) with ω = 0, τ = s (violet), and the pooled mean and variance of 64 real seeded dense layers (blue dots). Right: per-layer variance and mean for SELU, ReLU and tanh, with the same weights. At s = 1 (LeCun normal) from (1, 2) the map reaches (0.090, 1.714) after one layer and (0.007, 1.060) after ten; the real net sits at mean 0.057, variance 0.968 at layer 64, while ReLU's variance falls to 1.3e−20 and tanh's to 0.008. Move s: at 0.9 the attracting point shifts to (−0.033, 0.674), at 1.1 to (0.049, 1.399), and at s = 2 (He init) the real SELU net explodes.

Worked example: iterating the map

kSELU from (1, 2)ReLU from (0, 1)tanh from (0, 1)
0(1, 2)(0, 1)(0, 1)
1(0.090, 1.714)(0.399, 0.341)(0, 0.394)
2(0.064, 1.521)(0.233, 0.116)(0, 0.236)
5(0.026, 1.220)(0.046, 0.005)(0, 0.103)
10(0.007, 1.060)(0.003, 0.000)(0, 0.052)
64(0.000, 1.000)(0.000, 0.000)(0, 0.008)
(μ, ν) after k layers of the analytic map, ω = 0, τ = 1, starting from (1, 2); ReLU and tanh from (0, 1) for contrast (computed in Node)

The first step does most of the mean correction: with ω=0\omega=0 the input mean cannot leak through the weights, so only SELU's own asymmetry sets μ~\tilde\mu. The variance then decays geometrically toward 1. ReLU and tanh have no such attractor at τ=1\tau=1: their maps only shrink the variance (ReLU halves the second moment; tanh compresses it). ReLU needs τ=2\tau=2, He init, to hold the variance; SELU needs τ=1\tau=1 to hold it at exactly 1.

What it requires

requirementwhy
LeCun-normal init, w ~ N(0, 1/fan_in)gives ω≈0\omega\approx0, τ≈1\tau\approx1. He init has τ=2\tau=2, which moves the attracting point and, in the figure, makes the variance explode
AlphaDropout, not dropoutstandard dropout sets units to 0 and rescales the rest by 1/(1−p)1/(1-p), which changes the variance. AlphaDropout sets dropped units to SELU's floor −λα=−1.7581-\lambda\alpha=-1.7581 and applies an affine correction; for p=0.1p=0.1 that is a x+ba\,x+b with a=0.9213a=0.9213, b=0.1620b=0.1620
Standardised inputsthe argument assumes the first layer's inputs are near zero mean and unit variance
Plain dense stacksa residual sum x+F(x)x+F(x) of two unit-variance streams has variance near 2, and BatchNorm or LayerNorm re-impose their own statistics, so the fixed-point argument no longer describes the network
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3LAMBDA, ALPHA = 1.0507009873554805, 1.6732632423543772
4
5def selu(x): # from scratch (autograd handles the backward)
6 return LAMBDA * torch.where(x > 0, x, ALPHA * torch.expm1(x))
7
8x = torch.tensor([-3.0, -1.0, -0.5, 0.5, 2.0])
9print(F.selu(x)) # tensor([-1.6706, -1.1113, -0.6918, 0.5254, 2.1014])
10print(selu(x)) # same values; nn.SELU() is the module form
11
12def snn(d_in, width=256, depth=16, p=0.05):
13 layers = []
14 for i in range(depth):
15 lin = nn.Linear(d_in if i == 0 else width, width)
16 # LeCun normal: std = 1/sqrt(fan_in). Use "linear", NOT "selu" (that gain is 3/4)
17 nn.init.kaiming_normal_(lin.weight, nonlinearity="linear")
18 nn.init.zeros_(lin.bias)
19 layers += [lin, nn.SELU(), nn.AlphaDropout(p)]
20 return nn.Sequential(*layers, nn.Linear(width, 1))
21
22net = snn(32).eval() # eval(): AlphaDropout is the identity
23with torch.no_grad():
24 h = torch.randn(4096, 32) # standardised inputs
25 for m in net:
26 h = m(h)
27 if isinstance(m, nn.SELU):
28 print(f"mean {h.mean():+.3f} var {h.var():.3f}") # stays close to (0, 1)
A self-normalizing MLP in PyTorch, and SELU from scratch

Use when

  • +Normalization without normalization layers: no batch statistics, works at batch size 1
  • +Deep MLPs train stably; proven attracting fixed point at (0,1)
  • +No dead units; mean zero by construction
  • +Same cost as ELU, available everywhere (nn.SELU, AlphaDropout)

Avoid when

  • −Only valid with LeCun-normal init, AlphaDropout and standardised inputs
  • −Theory covers plain dense stacks, not residual or attention architectures
  • −Fixed point shifts with weight scale τ, and weights drift during training
  • −Rarely used in CNNs and transformers, where BatchNorm/LayerNorm dominate

Check yourself

A SELU network is initialised with He-normal weights, Var(w) = 2/fan_in, instead of LeCun normal. What happens to the activation variance through depth?

Check yourself

Why does SELU need AlphaDropout rather than ordinary dropout?

Remember

  • ✓SELU=λ ELUα\mathrm{SELU}=\lambda\,\mathrm{ELU}_\alpha with λ=1.0507\lambda=1.0507, α=1.6733\alpha=1.6733: the unique pair giving mean 0 and variance 1 for N(0,1)\mathcal N(0,1) input.
  • ✓The moment map (μ,ν)↦(μ~,ν~)(\mu,\nu)\mapsto(\tilde\mu,\tilde\nu) has a stable, attracting fixed point at (0,1)(0,1) for ω=0\omega=0, τ=1\tau=1 (Klambauer et al., 2017).
  • ✓Requires LeCun-normal init (Var⁡(w)=1/n\operatorname{Var}(w)=1/n), AlphaDropout and standardised inputs; He init or the "selu" gain break it.
  • ✓Works for plain deep MLPs; BatchNorm, residual sums and attention architectures fall outside the theory.
  • ✓λ>1\lambda>1 lets small variances grow and the saturating left branch caps large ones: both are needed for an attractor.
← back to Deep Learning roadmap