# SELU

> λ·ELU_α with λ ≈ 1.0507 and α ≈ 1.6733, chosen so that zero mean and unit variance are an attracting fixed point of every dense layer: with LeCun-normal weights the net normalises itself.

Source: https://www.mlfoundry.in/learn/selu  
Section: Deep Learning › Activation Functions  
Level: advanced · 10 min read · updated 2026-10-11

**SELU:** (scaled exponential linear unit) is $\mathrm{SELU}(x)=\lambda x$ for $x>0$ and $\lambda\alpha(e^{x}-1)$ for $x\le0$, with the fixed constants $\lambda\approx1.0507$ and $\alpha\approx1.6733$. Combined with LeCun-normal initialization it makes a deep feed-forward network **self-normalizing**: activation means and variances converge to $(0,1)$ layer by layer, without batch normalization.

Batch normalization keeps activations at zero mean and unit variance by measuring and correcting them. Klambauer, Unterthiner, Mayr and Hochreiter asked whether the activation function could do the job instead ("Self-Normalizing Neural Networks", NeurIPS 2017). They chose the two free numbers in a scaled ELU so that a layer maps zero mean and unit variance to zero mean and unit variance, and proved that nearby moments are pulled toward that point.

With it they trained fully connected networks with many layers without normalization layers. Self-normalizing networks (SNNs) beat the other feed-forward variants they compared (BatchNorm, LayerNorm, WeightNorm, highway and residual MLPs) on 121 UCI classification tasks, and did best on the Tox21 drug-toxicity benchmark and the HTRU2 pulsar dataset. Their best models were often deep, where most other MLP variants worked best shallow.

## Formula and derivative

$$
\mathrm{SELU}(x)=\lambda\begin{cases}x & x>0\\ \alpha\,(e^{x}-1) & x\le 0\end{cases}\qquad\qquad \mathrm{SELU}'(x)=\begin{cases}\lambda & x>0\\ \lambda\alpha\,e^{x}=\mathrm{SELU}(x)+\lambda\alpha & x\le 0\end{cases}
$$

- $\lambda=1.0507009873554805$: scale; $>1$, so the right branch can **increase** a variance that is too small
- $\alpha=1.6732632423543772$: ELU's negative saturation; the output floor is $-\lambda\alpha=-1.7581$
- $\lambda\alpha e^{x}$: left slope: $1.7581$ at $0^-$ against $1.0507$ at $0^+$, so SELU has a kink in its derivative

The constants are not tuned by experiment; they solve two equations. For $z\sim\mathcal N(0,1)$ require $\mathbb E[\mathrm{SELU}(z)]=0$ and $\mathbb E[\mathrm{SELU}(z)^2]=1$. The mean condition fixes $\alpha$ alone, because $\lambda$ multiplies everything:

$$
\alpha=\frac{1/\sqrt{2\pi}}{\tfrac12-e^{1/2}\Phi(-1)}=1.6733,\qquad \lambda=\Big(\tfrac12+\alpha^2\big(e^{2}\Phi(-2)-2e^{1/2}\Phi(-1)+\tfrac12\big)\Big)^{-1/2}=1.0507
$$

- $\Phi$: standard normal CDF: $e^{1/2}\Phi(-1)=0.26158$, $e^{2}\Phi(-2)=0.16810$
- $\tfrac12$: $\mathbb E[z^2\mathbb 1[z>0]]$, the positive half's second moment
- $\alpha^2(\cdots)$: $\alpha^2\,\mathbb E[(e^z-1)^2\mathbb 1[z<0]]=\alpha^2\cdot0.14495$, the negative half's

**Properties**

- **Range:** $(-\lambda\alpha,\infty)=(-1.7581,\infty)$
- **Monotonic:** yes, strictly increasing
- **Smooth:** continuous; the slope jumps from $1.7581$ to $1.0507$ at 0
- **Zero-centred:** by construction: mean exactly 0 and variance exactly 1 for $\mathcal N(0,1)$ input
- **Saturating:** softly on the left (toward $-1.7581$), never on the right
- **Requirements:** LeCun-normal init, AlphaDropout instead of dropout, plain dense layers, (0,1)-standardised inputs
- **Cost:** ELU's: one exp per negative input

> Figure: SELU against ELU (α = 1) and ReLU. Note the floor at −1.758, the slope of 1.0507 on the right and the jump in the derivative at 0. For standard normal inputs the statistics panel shows SELU's output mean 0 and variance 1.0000, the property everything else is built on.

## The moment map and its fixed point

Consider one unit whose $n$ inputs have mean $\mu$ and variance $\nu$, with weights summarised by $\omega=\sum_i w_i$ and $\tau=\sum_i w_i^2$. By the central limit theorem its pre-activation is approximately $\mathcal N(\mu\omega,\nu\tau)$, and the output's moments are a deterministic function of the input's:

$$
\tilde\mu=\mathbb E\big[\mathrm{SELU}(z)\big],\qquad \tilde\nu=\mathbb E\big[\mathrm{SELU}(z)^2\big]-\tilde\mu^2,\qquad z\sim\mathcal N(\mu\omega,\;\nu\tau)
$$

- $(\mu,\nu)\mapsto(\tilde\mu,\tilde\nu)$: the map g applied once per layer; a deep net iterates it
- $\omega=0,\ \tau=1$: normalized weights; LeCun normal $w\sim\mathcal N(0,1/n)$ gives this in expectation
- $(0,1)$: a fixed point of $g$ by the choice of $\lambda,\alpha$

Theorem 1 of the paper shows that $(0,1)$ is a **stable and attracting** fixed point: $g$ is a contraction on a neighbourhood of it (the theorem's domain is $\mu,\omega\in[-0.1,0.1]$, $\nu\in[0.8,1.5]$, $\tau\in[0.95,1.1]$). Two further theorems bound the variance from above and below, so it can neither explode nor vanish even starting outside that region. The mechanism is visible in the formula: when $\nu$ is small, the inputs mostly use the right branch with slope $\lambda>1$, which raises the variance; when $\nu$ is large, more inputs land on the saturating left branch, which caps it.

> Figure: Left: SELU's moment map iterated from the input (μ₀, ν₀) with ω = 0, τ = s (violet), and the pooled mean and variance of 64 real seeded dense layers (blue dots). Right: per-layer variance and mean for SELU, ReLU and tanh, with the same weights. At s = 1 (LeCun normal) from (1, 2) the map reaches (0.090, 1.714) after one layer and (0.007, 1.060) after ten; the real net sits at mean 0.057, variance 0.968 at layer 64, while ReLU's variance falls to 1.3e−20 and tanh's to 0.008. Move s: at 0.9 the attracting point shifts to (−0.033, 0.674), at 1.1 to (0.049, 1.399), and at s = 2 (He init) the real SELU net explodes.

## Worked example: iterating the map

(μ, ν) after k layers of the analytic map, ω = 0, τ = 1, starting from (1, 2); ReLU and tanh from (0, 1) for contrast (computed in Node)

| k | SELU from (1, 2) | ReLU from (0, 1) | tanh from (0, 1) |
| --- | --- | --- | --- |
| 0 | (1, 2) | (0, 1) | (0, 1) |
| 1 | (0.090, 1.714) | (0.399, 0.341) | (0, 0.394) |
| 2 | (0.064, 1.521) | (0.233, 0.116) | (0, 0.236) |
| 5 | (0.026, 1.220) | (0.046, 0.005) | (0, 0.103) |
| 10 | (0.007, 1.060) | (0.003, 0.000) | (0, 0.052) |
| 64 | (0.000, 1.000) | (0.000, 0.000) | (0, 0.008) |

The first step does most of the mean correction: with $\omega=0$ the input mean cannot leak through the weights, so only SELU's own asymmetry sets $\tilde\mu$. The variance then decays geometrically toward 1. ReLU and tanh have no such attractor at $\tau=1$: their maps only shrink the variance (ReLU halves the second moment; tanh compresses it). ReLU needs $\tau=2$, He init, to hold the variance; SELU needs $\tau=1$ to hold it at exactly 1.

## What it requires

| requirement | why |
| --- | --- |
| **LeCun-normal init**, w ~ N(0, 1/fan_in) | gives $\omega\approx0$, $\tau\approx1$. He init has $\tau=2$, which moves the attracting point and, in the figure, makes the variance explode |
| **AlphaDropout**, not dropout | standard dropout sets units to 0 and rescales the rest by $1/(1-p)$, which changes the variance. AlphaDropout sets dropped units to SELU's floor $-\lambda\alpha=-1.7581$ and applies an affine correction; for $p=0.1$ that is $a\,x+b$ with $a=0.9213$, $b=0.1620$ |
| **Standardised inputs** | the argument assumes the first layer's inputs are near zero mean and unit variance |
| **Plain dense stacks** | a residual sum $x+F(x)$ of two unit-variance streams has variance near 2, and BatchNorm or LayerNorm re-impose their own statistics, so the fixed-point argument no longer describes the network |

```python
import torch, torch.nn as nn, torch.nn.functional as F

LAMBDA, ALPHA = 1.0507009873554805, 1.6732632423543772

def selu(x):                                   # from scratch (autograd handles the backward)
    return LAMBDA * torch.where(x > 0, x, ALPHA * torch.expm1(x))

x = torch.tensor([-3.0, -1.0, -0.5, 0.5, 2.0])
print(F.selu(x))     # tensor([-1.6706, -1.1113, -0.6918,  0.5254,  2.1014])
print(selu(x))       # same values;  nn.SELU() is the module form

def snn(d_in, width=256, depth=16, p=0.05):
    layers = []
    for i in range(depth):
        lin = nn.Linear(d_in if i == 0 else width, width)
        # LeCun normal: std = 1/sqrt(fan_in). Use "linear", NOT "selu" (that gain is 3/4)
        nn.init.kaiming_normal_(lin.weight, nonlinearity="linear")
        nn.init.zeros_(lin.bias)
        layers += [lin, nn.SELU(), nn.AlphaDropout(p)]
    return nn.Sequential(*layers, nn.Linear(width, 1))

net = snn(32).eval()                           # eval(): AlphaDropout is the identity
with torch.no_grad():
    h = torch.randn(4096, 32)                  # standardised inputs
    for m in net:
        h = m(h)
        if isinstance(m, nn.SELU):
            print(f"mean {h.mean():+.3f}  var {h.var():.3f}")   # stays close to (0, 1)
```

A self-normalizing MLP in PyTorch, and SELU from scratch

> **The selu gain trap:** `torch.nn.init.calculate_gain("selu")` returns $3/4$, which gives weights with variance $\tfrac{9}{16}\cdot\tfrac1n$, not $\tfrac1n$. PyTorch's own documentation says to use `nonlinearity="linear"` for self-normalizing networks. Likewise Keras pairs `selu` with `lecun_normal`, not the default Glorot initializer.

> **Mixing breaks the guarantee:** Add BatchNorm, residual connections, standard dropout or He init to a SELU network and it still trains, but it is no longer self-normalizing: the fixed point the theorems describe is gone, and you are back to tuning normalization yourself. SELU earns its keep only as a complete recipe.

> **Why λ > 1:** If the scale were 1 (plain ELU), the map could only shrink the variance. SELU's positive branch has slope $1.0507$, so a layer with small input variance, whose inputs are mostly in the near-linear region, amplifies it. The saturating negative branch caps large variances. An attracting fixed point needs both: a way up and a way down.

**Strengths**

- Normalization without normalization layers: no batch statistics, works at batch size 1
- Deep MLPs train stably; proven attracting fixed point at (0,1)
- No dead units; mean zero by construction
- Same cost as ELU, available everywhere (nn.SELU, AlphaDropout)

**Limitations**

- Only valid with LeCun-normal init, AlphaDropout and standardised inputs
- Theory covers plain dense stacks, not residual or attention architectures
- Fixed point shifts with weight scale τ, and weights drift during training
- Rarely used in CNNs and transformers, where BatchNorm/LayerNorm dominate

**Question:** A SELU network is initialised with He-normal weights, Var(w) = 2/fan_in, instead of LeCun normal. What happens to the activation variance through depth?

- A. It still converges to 1
- B. It converges to 0
- C. It grows: τ = 2 is outside the regime where (0, 1) is attracting
- D. Nothing changes, because SELU ignores the weight scale

**Answer:** C. It grows: τ = 2 is outside the regime where (0, 1) is attracting — The fixed point $(0,1)$ was built for $\tau=\sum w_i^2\approx1$. With $\tau=2$ the map pushes the variance up; in the figure the real 64-layer SELU net reaches variances in the hundreds at $s=2$. Self-normalization is a property of the activation and the init together.

**Question:** Why does SELU need AlphaDropout rather than ordinary dropout?

- A. Ordinary dropout is too slow
- B. Ordinary dropout sets units to 0 and rescales the rest, which changes the mean and variance SELU is holding at (0, 1)
- C. SELU cannot be combined with any regularization
- D. AlphaDropout drops whole layers

**Answer:** B. Ordinary dropout sets units to 0 and rescales the rest, which changes the mean and variance SELU is holding at (0, 1) — Zero is not SELU's resting value, and the $1/(1-p)$ rescaling inflates the variance. AlphaDropout sets dropped units to the saturation value $-\lambda\alpha$ and then applies an affine map $ax+b$ chosen so that the mean and variance are unchanged.

**Key points**

- $\mathrm{SELU}=\lambda\,\mathrm{ELU}_\alpha$ with $\lambda=1.0507$, $\alpha=1.6733$: the unique pair giving mean 0 and variance 1 for $\mathcal N(0,1)$ input.
- The moment map $(\mu,\nu)\mapsto(\tilde\mu,\tilde\nu)$ has a stable, attracting fixed point at $(0,1)$ for $\omega=0$, $\tau=1$ (Klambauer et al., 2017).
- Requires LeCun-normal init ($\operatorname{Var}(w)=1/n$), AlphaDropout and standardised inputs; He init or the "selu" gain break it.
- Works for plain deep MLPs; BatchNorm, residual sums and attention architectures fall outside the theory.
- $\lambda>1$ lets small variances grow and the saturating left branch caps large ones: both are needed for an attractor.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
