# ELU

> x for x > 0, α(eˣ − 1) below: negative outputs pull the mean activation toward zero, the curve is smooth at 0 when α = 1, and it saturates softly at −α.

Source: https://www.mlfoundry.in/learn/elu  
Section: Deep Learning › Activation Functions  
Level: core · 9 min read · updated 2026-10-11

**ELU:** (exponential linear unit) is the activation $\mathrm{ELU}_\alpha(x)=x$ for $x>0$ and $\alpha(e^{x}-1)$ for $x\le0$, with $\alpha>0$ (usually 1). It is the identity on the right and an exponential that saturates at $-\alpha$ on the left.

ReLU's outputs are never negative, so a layer of ReLUs always has a positive mean. ELU keeps the identity on the positive side, so it does not saturate there, and replaces the flat zero with a smooth curve into negative values. Negative outputs pull the mean toward zero, the gradient on the left is never exactly zero, and with $\alpha=1$ the curve has no kink.

Clevert, Unterthiner and Hochreiter introduced ELU in 2015 ("Fast and Accurate Deep Network Learning by Exponential Linear Units", ICLR 2016). They reported faster learning and better generalization than ReLU and Leaky ReLU in networks with more than 5 layers, and the best published CIFAR-100 result at the time without multi-view evaluation or model averaging. ELU is also the parent of **SELU**, which rescales it so that a network normalises itself.

## Formula and derivative

$$
\mathrm{ELU}_\alpha(x)=\begin{cases}x & x>0\\ \alpha\,(e^{x}-1) & x\le 0\end{cases}\qquad\qquad \mathrm{ELU}_\alpha'(x)=\begin{cases}1 & x>0\\ \alpha\,e^{x}=\mathrm{ELU}_\alpha(x)+\alpha & x\le 0\end{cases}
$$

- $\alpha$: the negative saturation level: $\mathrm{ELU}(x)\to-\alpha$ as $x\to-\infty$
- $\alpha e^{x}$: slope on the left: $\alpha$ at $0^-$, decaying to 0
- $\mathrm{ELU}(x)+\alpha$: the backward pass can reuse the forward output instead of recomputing the exponential

At $0$ the left slope is $\alpha$ and the right slope is $1$. So ELU is continuously differentiable ($C^1$) **only for $\alpha=1$**; any other $\alpha$ leaves a kink in the derivative. The second derivative jumps at 0 even for $\alpha=1$ (from $1$ to $0$). **CELU** (Barron, 2017), $\max(0,x)+\min\big(0,\alpha(e^{x/\alpha}-1)\big)$, rescales the input of the exponential so that the slope at $0^-$ is 1 for every $\alpha$.

**Properties**

- **Range:** $(-\alpha,\infty)$
- **Monotonic:** yes, strictly increasing
- **Smooth:** $C^1$ when $\alpha=1$; otherwise the slope jumps from $\alpha$ to 1 at 0
- **Zero-centred:** closer than ReLU: for $\mathcal N(0,1)$ input the mean is $0.161$ ($\alpha=1$) vs ReLU's $0.399$, and exactly 0 at $\alpha=1.6733$
- **Saturating:** softly on the left only, toward $-\alpha$; never on the right
- **Gradient:** never exactly 0, but $\alpha e^{x}$ is tiny for very negative $x$ ($0.0498$ at $x=-3$)
- **Cost:** one exp per negative input forward; the backward reuses the output

> Figure: ELU (α = 1) against ReLU and Leaky ReLU. Drag α: the floor at −α moves, and the derivative on the left starts at α, so only α = 1 joins the right-hand slope without a jump. The output statistics show the mean falling as α grows.

## Pulling the mean toward zero

For a unit whose input is $z\sim\mathcal N(0,1)$, ELU's negative half contributes $\alpha\,\mathbb E[(e^{z}-1)\mathbb 1[z<0]]=\alpha(e^{1/2}\Phi(-1)-\tfrac12)=-0.2384\,\alpha$ to the mean, which offsets the positive half's $1/\sqrt{2\pi}=0.3989$:

$$
\mathbb E\big[\mathrm{ELU}_\alpha(z)\big]=\frac{1}{\sqrt{2\pi}}-\alpha\Big(\tfrac12-e^{1/2}\,\Phi(-1)\Big)=0.3989-0.2384\,\alpha,\qquad z\sim\mathcal N(0,1)
$$

- $\Phi$: standard normal CDF; $e^{1/2}\Phi(-1)=0.2616$
- $\alpha=1$: mean $0.161$
- $\alpha=1.6733$: mean exactly 0: the same constant that appears in SELU

Why care about the mean? A positive mean in layer $l$ acts like a bias on every unit of layer $l+1$, so the next layer's inputs drift with every update to the weights below (Clevert et al. call this **bias shift**). Their argument, in the spirit of natural-gradient methods, is that activations with means closer to zero make the ordinary gradient closer to the natural gradient, and so speed up learning. Batch normalization attacks the same problem by subtracting the batch mean. ELU gets part of the way there for the cost of an exponential.

The soft floor at $-\alpha$ matters too. Very negative inputs all map to roughly $-\alpha$, so the "off" state carries little information about how negative the input was. The paper calls this a noise-robust deactivation state, in contrast with Leaky ReLU, whose negative side is linear and unbounded.

> Figure: A 30-layer stack of 48-unit dense layers (no biases, one seeded weight draw shared by both nets) on 96 Gaussian inputs. Left: the bias shift, mean ÷ rms of each layer's activations. Right: their spread. With variance-matched weights (ReLU s = √2, ELU s = 1.245 at α = 1) ReLU's bias shift averages 0.548 over the 30 layers and ELU's 0.177; push α to 1.5 and it averages 0.032. Switch ELU to He init (s = √2): the mean share rises to 0.304 and the spread grows from 1.05 at layer 1 to 1.78 at layer 30, because He is calibrated for ReLU. The histogram at layer 20 shows ReLU's spike of exact zeros (53%) next to ELU's mass piled up above −α.

## Initialization

ELU passes more of the second moment than ReLU: for $z\sim\mathcal N(0,1)$, $\mathbb E[\mathrm{ELU}_1(z)^2]=0.645$ against ReLU's $0.5$. He initialization ($\operatorname{Var}(w)=2/n$) therefore slightly over-amplifies an ELU network, as the figure shows. The variance-preserving gain is $1/\sqrt{0.645}=1.245$ for $\alpha=1$. In practice ELU nets are usually trained with He init plus normalization, which absorbs the difference; PyTorch's `calculate_gain` has no ELU entry.

## Worked example

ELU with α = 1, ReLU for comparison (computed in Node)

| x | ELU(x) | ELU′(x) | check: ELU(x) + α | ReLU(x) | ReLU′(x) |
| --- | --- | --- | --- | --- | --- |
| −3 | −0.9502 | 0.0498 | 0.0498 | 0 | 0 |
| −1 | −0.6321 | 0.3679 | 0.3679 | 0 | 0 |
| −0.5 | −0.3935 | 0.6065 | 0.6065 | 0 | 0 |
| 0 | 0 | 1 (both sides, α = 1) | 1 | 0 | 0 (convention) |
| 0.5 | 0.5 | 1 | — | 0.5 | 1 |
| 2 | 2 | 1 | — | 2 | 1 |

**One unit, x = −1, upstream gradient δ = 0.5**

1. Forward: $\mathrm{ELU}(-1)=e^{-1}-1=-0.6321$. Store the output.
2. Backward: $\mathrm{ELU}'(-1)=-0.6321+1=0.3679$, no second exponential needed.
3. The unit passes $0.5\cdot0.3679=0.1839$ back. A ReLU unit at the same input would pass 0; Leaky ReLU with $\alpha=0.01$ would pass $0.005$.
4. At $x=-5$ the ELU factor is $e^{-5}=0.0067$: very negative units saturate, so ELU softens the dying problem rather than removing it.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-3.0, -1.0, -0.5, 0.5, 2.0], requires_grad=True)

y = nn.ELU(alpha=1.0)(x)      # module; nn.CELU(alpha) is the C^1-for-every-alpha variant
y = F.elu(x, alpha=1.0)       # functional
y.sum().backward()
print(y.detach())   # tensor([-0.9502, -0.6321, -0.3935,  0.5000,  2.0000])
print(x.grad)       # tensor([0.0498, 0.3679, 0.6065, 1.0000, 1.0000])

class MyELU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, alpha=1.0):
        y = torch.where(x > 0, x, alpha * torch.expm1(x))   # expm1 is accurate near 0
        ctx.save_for_backward(x, y)
        ctx.alpha = alpha
        return y
    @staticmethod
    def backward(ctx, g):
        x, y = ctx.saved_tensors
        return g * torch.where(x > 0, torch.ones_like(y), y + ctx.alpha), None
```

ELU in PyTorch, plus a from-scratch version whose backward reuses the output

> **α ≠ 1 brings the kink back:** The smoothness argument holds only at $\alpha=1$: at $\alpha=2$ the slope jumps from 2 to 1 at the origin. Large $\alpha$ also makes the negative side steeper than the positive one and the floor deeper, which changes the variance the init has to match. If you want a different floor but a smooth curve, use CELU.

> **From ELU to SELU:** The mean formula above hits zero at $\alpha=1.6733$, and multiplying by $\lambda=1.0507$ then makes the variance exactly 1 for $\mathcal N(0,1)$ input. Those are precisely SELU's constants: SELU is the ELU scaled so that zero mean and unit variance are a fixed point of every layer.

> **Where you meet it:** ELU is available in every framework and shows up in some GAN, VAE and graph-network code, but it never displaced ReLU in vision backbones, and transformers moved to GELU and SiLU variants. Its main legacy is the idea that a smooth negative branch can stand in for part of what normalization does.

**Strengths**

- Mean activation closer to zero than ReLU (less bias shift)
- Identity on the right: no saturation for positive inputs
- Gradient never exactly zero; smooth at 0 when α = 1
- Bounded below: a noise-robust off state

**Limitations**

- An exp per negative input: slower than ReLU
- Still saturates (softly) for very negative inputs
- He init over-amplifies it slightly; needs a matched gain or normalization
- Smoothness only at α = 1; not sparse (no exact zeros)

**Question:** For which value of α is ELU continuously differentiable at 0?

- A. Any α > 0
- B. α = 1 only
- C. α = 0
- D. α = 1.6733

**Answer:** B. α = 1 only — The left derivative at $0$ is $\alpha e^{0}=\alpha$ and the right derivative is 1, so they agree only when $\alpha=1$. $\alpha=1.6733$ is the value that makes the mean zero for standard normal inputs, not the smooth one.

**Question:** An ELU unit (α = 1) output y = −0.8 during the forward pass. What is its local derivative, without recomputing any exponential?

- A. 0
- B. 0.2
- C. 0.8
- D. 1

**Answer:** B. 0.2 — For $x\le0$, $\mathrm{ELU}'(x)=\alpha e^{x}=\mathrm{ELU}(x)+\alpha=-0.8+1=0.2$. Implementations save the output and use exactly this identity in the backward pass.

**Key points**

- $\mathrm{ELU}_\alpha(x)=x$ for $x>0$, $\alpha(e^x-1)$ for $x\le0$; derivative 1 or $\alpha e^x=\mathrm{ELU}(x)+\alpha$.
- Negative outputs pull the mean toward zero: $0.399-0.238\alpha$ for standard normal input, vs ReLU's $0.399$.
- Smooth ($C^1$) only at $\alpha=1$; saturates softly at $-\alpha$; costs an exponential.
- Clevert et al. (2015) argued a near-zero mean reduces bias shift and brings the gradient closer to the natural gradient.
- Scaling ELU with $\alpha=1.6733$ by $\lambda=1.0507$ gives SELU.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
