ELU

x for x > 0, α(eˣ − 1) below: negative outputs pull the mean activation toward zero, the curve is smooth at 0 when α = 1, and it saturates softly at −α.

9 min read734 wordscore0 viewsupdated 2026-10-11

Definition

ELU (exponential linear unit) is the activation ELUα(x)=x\mathrm{ELU}_\alpha(x)=x for x>0x>0 and α(ex−1)\alpha(e^{x}-1) for x≤0x\le0, with α>0\alpha>0 (usually 1). It is the identity on the right and an exponential that saturates at −α-\alpha on the left.

ReLU's outputs are never negative, so a layer of ReLUs always has a positive mean. ELU keeps the identity on the positive side, so it does not saturate there, and replaces the flat zero with a smooth curve into negative values. Negative outputs pull the mean toward zero, the gradient on the left is never exactly zero, and with α=1\alpha=1 the curve has no kink.

Clevert, Unterthiner and Hochreiter introduced ELU in 2015 ("Fast and Accurate Deep Network Learning by Exponential Linear Units", ICLR 2016). They reported faster learning and better generalization than ReLU and Leaky ReLU in networks with more than 5 layers, and the best published CIFAR-100 result at the time without multi-view evaluation or model averaging. ELU is also the parent of SELU, which rescales it so that a network normalises itself.

Formula and derivative

ELUα(x)={xx>0α (ex−1)x≤0ELUα′(x)={1x>0α ex=ELUα(x)+αx≤0\mathrm{ELU}_\alpha(x)=\begin{cases}x & x>0\\ \alpha\,(e^{x}-1) & x\le 0\end{cases}\qquad\qquad \mathrm{ELU}_\alpha'(x)=\begin{cases}1 & x>0\\ \alpha\,e^{x}=\mathrm{ELU}_\alpha(x)+\alpha & x\le 0\end{cases}
α\alpha
the negative saturation level: ELU(x)→−α\mathrm{ELU}(x)\to-\alpha as x→−∞x\to-\infty
αex\alpha e^{x}
slope on the left: α\alpha at 0−0^-, decaying to 0
ELU(x)+α\mathrm{ELU}(x)+\alpha
the backward pass can reuse the forward output instead of recomputing the exponential

At 00 the left slope is α\alpha and the right slope is 11. So ELU is continuously differentiable (C1C^1) only for α=1\alpha=1; any other α\alpha leaves a kink in the derivative. The second derivative jumps at 0 even for α=1\alpha=1 (from 11 to 00). CELU (Barron, 2017), max⁡(0,x)+min⁡(0,α(ex/α−1))\max(0,x)+\min\big(0,\alpha(e^{x/\alpha}-1)\big), rescales the input of the exponential so that the slope at 0−0^- is 1 for every α\alpha.

Properties

Range
(−α,∞)(-\alpha,\infty)
Monotonic
yes, strictly increasing
Smooth
C1C^1 when α=1\alpha=1; otherwise the slope jumps from α\alpha to 1 at 0
Zero-centred
closer than ReLU: for N(0,1)\mathcal N(0,1) input the mean is 0.1610.161 (α=1\alpha=1) vs ReLU's 0.3990.399, and exactly 0 at α=1.6733\alpha=1.6733
Saturating
softly on the left only, toward −α-\alpha; never on the right
Gradient
never exactly 0, but αex\alpha e^{x} is tiny for very negative xx (0.04980.0498 at x=−3x=-3)
Cost
one exp per negative input forward; the backward reuses the output
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 1.000 · f′(x) = 1.000−2024−4−2024x →ELU (α=1)ELU f′ReLULeaky ReLU
activation
compare
range (−α, ∞)monotonic yessmooth nozero-centred no
C¹ only when α = 1 (otherwise a kink at 0). Negative saturation at −α pulls mean activations toward zero.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]−0.981.514.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.161Var[f(x)] 0.619E[f′(x)] 0.762f′ = 0 for 0.0% of inputsE[f′]¹⁰ 0.066
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.ELU (α = 1) against ReLU and Leaky ReLU. Drag α: the floor at −α moves, and the derivative on the left starts at α, so only α = 1 joins the right-hand slope without a jump. The output statistics show the mean falling as α grows.

Pulling the mean toward zero

For a unit whose input is z∼N(0,1)z\sim\mathcal N(0,1), ELU's negative half contributes α E[(ez−1)1[z<0]]=α(e1/2Φ(−1)−12)=−0.2384 α\alpha\,\mathbb E[(e^{z}-1)\mathbb 1[z<0]]=\alpha(e^{1/2}\Phi(-1)-\tfrac12)=-0.2384\,\alpha to the mean, which offsets the positive half's 1/2π=0.39891/\sqrt{2\pi}=0.3989:

E[ELUα(z)]=12π−α(12−e1/2 Φ(−1))=0.3989−0.2384 α,z∼N(0,1)\mathbb E\big[\mathrm{ELU}_\alpha(z)\big]=\frac{1}{\sqrt{2\pi}}-\alpha\Big(\tfrac12-e^{1/2}\,\Phi(-1)\Big)=0.3989-0.2384\,\alpha,\qquad z\sim\mathcal N(0,1)
Φ\Phi
standard normal CDF; e1/2Φ(−1)=0.2616e^{1/2}\Phi(-1)=0.2616
α=1\alpha=1
mean 0.1610.161
α=1.6733\alpha=1.6733
mean exactly 0: the same constant that appears in SELU

Why care about the mean? A positive mean in layer ll acts like a bias on every unit of layer l+1l+1, so the next layer's inputs drift with every update to the weights below (Clevert et al. call this bias shift). Their argument, in the spirit of natural-gradient methods, is that activations with means closer to zero make the ordinary gradient closer to the natural gradient, and so speed up learning. Batch normalization attacks the same problem by subtracting the batch mean. ELU gets part of the way there for the cost of an exponential.

The soft floor at −α-\alpha matters too. Very negative inputs all map to roughly −α-\alpha, so the "off" state carries little information about how negative the input was. The paper calls this a noise-robust deactivation state, in contrast with Leaky ReLU, whose negative side is linear and unbounded.

bias shift: mean / rms of activations−0.6−0.30.00.30.6std of activations per layer (log)1e-31e-21e-111011101020203030layerlayeractivations at layer 20 (96 inputs × 48 units, no biases, one weight draw)−α↑ 53% exactly 0−3−2−1012345━ ReLU layer 20: mean 0.249, std 0.411, mean/rms 0.52, s = 1.414━ ELU α=1.00 layer 20: mean 0.167, std 0.663, mean/rms 0.24, s = 1.245
W ~ N(0, s²/48)
fig.A 30-layer stack of 48-unit dense layers (no biases, one seeded weight draw shared by both nets) on 96 Gaussian inputs. Left: the bias shift, mean ÷ rms of each layer's activations. Right: their spread. With variance-matched weights (ReLU s = √2, ELU s = 1.245 at α = 1) ReLU's bias shift averages 0.548 over the 30 layers and ELU's 0.177; push α to 1.5 and it averages 0.032. Switch ELU to He init (s = √2): the mean share rises to 0.304 and the spread grows from 1.05 at layer 1 to 1.78 at layer 30, because He is calibrated for ReLU. The histogram at layer 20 shows ReLU's spike of exact zeros (53%) next to ELU's mass piled up above −α.

Initialization

ELU passes more of the second moment than ReLU: for z∼N(0,1)z\sim\mathcal N(0,1), E[ELU1(z)2]=0.645\mathbb E[\mathrm{ELU}_1(z)^2]=0.645 against ReLU's 0.50.5. He initialization (Var⁡(w)=2/n\operatorname{Var}(w)=2/n) therefore slightly over-amplifies an ELU network, as the figure shows. The variance-preserving gain is 1/0.645=1.2451/\sqrt{0.645}=1.245 for α=1\alpha=1. In practice ELU nets are usually trained with He init plus normalization, which absorbs the difference; PyTorch's calculate_gain has no ELU entry.

Worked example

xELU(x)ELU′(x)check: ELU(x) + αReLU(x)ReLU′(x)
−3−0.95020.04980.049800
−1−0.63210.36790.367900
−0.5−0.39350.60650.606500
001 (both sides, α = 1)100 (convention)
0.50.51—0.51
221—21
ELU with α = 1, ReLU for comparison (computed in Node)

One unit, x = −1, upstream gradient δ = 0.5

  1. 1Forward: ELU(−1)=e−1−1=−0.6321\mathrm{ELU}(-1)=e^{-1}-1=-0.6321. Store the output.
  2. 2Backward: ELU′(−1)=−0.6321+1=0.3679\mathrm{ELU}'(-1)=-0.6321+1=0.3679, no second exponential needed.
  3. 3The unit passes 0.5⋅0.3679=0.18390.5\cdot0.3679=0.1839 back. A ReLU unit at the same input would pass 0; Leaky ReLU with α=0.01\alpha=0.01 would pass 0.0050.005.
  4. 4At x=−5x=-5 the ELU factor is e−5=0.0067e^{-5}=0.0067: very negative units saturate, so ELU softens the dying problem rather than removing it.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-3.0, -1.0, -0.5, 0.5, 2.0], requires_grad=True)
4
5y = nn.ELU(alpha=1.0)(x) # module; nn.CELU(alpha) is the C^1-for-every-alpha variant
6y = F.elu(x, alpha=1.0) # functional
7y.sum().backward()
8print(y.detach()) # tensor([-0.9502, -0.6321, -0.3935, 0.5000, 2.0000])
9print(x.grad) # tensor([0.0498, 0.3679, 0.6065, 1.0000, 1.0000])
10
11class MyELU(torch.autograd.Function):
12 @staticmethod
13 def forward(ctx, x, alpha=1.0):
14 y = torch.where(x > 0, x, alpha * torch.expm1(x)) # expm1 is accurate near 0
15 ctx.save_for_backward(x, y)
16 ctx.alpha = alpha
17 return y
18 @staticmethod
19 def backward(ctx, g):
20 x, y = ctx.saved_tensors
21 return g * torch.where(x > 0, torch.ones_like(y), y + ctx.alpha), None
ELU in PyTorch, plus a from-scratch version whose backward reuses the output

Use when

  • +Mean activation closer to zero than ReLU (less bias shift)
  • +Identity on the right: no saturation for positive inputs
  • +Gradient never exactly zero; smooth at 0 when α = 1
  • +Bounded below: a noise-robust off state

Avoid when

  • −An exp per negative input: slower than ReLU
  • −Still saturates (softly) for very negative inputs
  • −He init over-amplifies it slightly; needs a matched gain or normalization
  • −Smoothness only at α = 1; not sparse (no exact zeros)

Check yourself

For which value of α is ELU continuously differentiable at 0?

Check yourself

An ELU unit (α = 1) output y = −0.8 during the forward pass. What is its local derivative, without recomputing any exponential?

Remember

  • ✓ELUα(x)=x\mathrm{ELU}_\alpha(x)=x for x>0x>0, α(ex−1)\alpha(e^x-1) for x≤0x\le0; derivative 1 or αex=ELU(x)+α\alpha e^x=\mathrm{ELU}(x)+\alpha.
  • ✓Negative outputs pull the mean toward zero: 0.399−0.238α0.399-0.238\alpha for standard normal input, vs ReLU's 0.3990.399.
  • ✓Smooth (C1C^1) only at α=1\alpha=1; saturates softly at −α-\alpha; costs an exponential.
  • ✓Clevert et al. (2015) argued a near-zero mean reduces bias shift and brings the gradient closer to the natural gradient.
  • ✓Scaling ELU with α=1.6733\alpha=1.6733 by λ=1.0507\lambda=1.0507 gives SELU.
← back to Deep Learning roadmap