Definition
ELU (exponential linear unit) is the activation for and for , with (usually 1). It is the identity on the right and an exponential that saturates at on the left.
ReLU's outputs are never negative, so a layer of ReLUs always has a positive mean. ELU keeps the identity on the positive side, so it does not saturate there, and replaces the flat zero with a smooth curve into negative values. Negative outputs pull the mean toward zero, the gradient on the left is never exactly zero, and with the curve has no kink.
Clevert, Unterthiner and Hochreiter introduced ELU in 2015 ("Fast and Accurate Deep Network Learning by Exponential Linear Units", ICLR 2016). They reported faster learning and better generalization than ReLU and Leaky ReLU in networks with more than 5 layers, and the best published CIFAR-100 result at the time without multi-view evaluation or model averaging. ELU is also the parent of SELU, which rescales it so that a network normalises itself.
Formula and derivative
- the negative saturation level: as
- slope on the left: at , decaying to 0
- the backward pass can reuse the forward output instead of recomputing the exponential
At the left slope is and the right slope is . So ELU is continuously differentiable () only for ; any other leaves a kink in the derivative. The second derivative jumps at 0 even for (from to ). CELU (Barron, 2017), , rescales the input of the exponential so that the slope at is 1 for every .
Properties
- Range
- Monotonic
- yes, strictly increasing
- Smooth
- when ; otherwise the slope jumps from to 1 at 0
- Zero-centred
- closer than ReLU: for input the mean is () vs ReLU's , and exactly 0 at
- Saturating
- softly on the left only, toward ; never on the right
- Gradient
- never exactly 0, but is tiny for very negative ( at )
- Cost
- one exp per negative input forward; the backward reuses the output
Pulling the mean toward zero
For a unit whose input is , ELU's negative half contributes to the mean, which offsets the positive half's :
- standard normal CDF;
- mean
- mean exactly 0: the same constant that appears in SELU
Why care about the mean? A positive mean in layer acts like a bias on every unit of layer , so the next layer's inputs drift with every update to the weights below (Clevert et al. call this bias shift). Their argument, in the spirit of natural-gradient methods, is that activations with means closer to zero make the ordinary gradient closer to the natural gradient, and so speed up learning. Batch normalization attacks the same problem by subtracting the batch mean. ELU gets part of the way there for the cost of an exponential.
The soft floor at matters too. Very negative inputs all map to roughly , so the "off" state carries little information about how negative the input was. The paper calls this a noise-robust deactivation state, in contrast with Leaky ReLU, whose negative side is linear and unbounded.
Initialization
ELU passes more of the second moment than ReLU: for , against ReLU's . He initialization () therefore slightly over-amplifies an ELU network, as the figure shows. The variance-preserving gain is for . In practice ELU nets are usually trained with He init plus normalization, which absorbs the difference; PyTorch's calculate_gain has no ELU entry.
Worked example
| x | ELU(x) | ELU′(x) | check: ELU(x) + α | ReLU(x) | ReLU′(x) |
|---|---|---|---|---|---|
| −3 | −0.9502 | 0.0498 | 0.0498 | 0 | 0 |
| −1 | −0.6321 | 0.3679 | 0.3679 | 0 | 0 |
| −0.5 | −0.3935 | 0.6065 | 0.6065 | 0 | 0 |
| 0 | 0 | 1 (both sides, α = 1) | 1 | 0 | 0 (convention) |
| 0.5 | 0.5 | 1 | — | 0.5 | 1 |
| 2 | 2 | 1 | — | 2 | 1 |
One unit, x = −1, upstream gradient δ = 0.5
- 1Forward: . Store the output.
- 2Backward: , no second exponential needed.
- 3The unit passes back. A ReLU unit at the same input would pass 0; Leaky ReLU with would pass .
- 4At the ELU factor is : very negative units saturate, so ELU softens the dying problem rather than removing it.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([-3.0, -1.0, -0.5, 0.5, 2.0], requires_grad=True)45y = nn.ELU(alpha=1.0)(x) # module; nn.CELU(alpha) is the C^1-for-every-alpha variant6y = F.elu(x, alpha=1.0) # functional7y.sum().backward()8print(y.detach()) # tensor([-0.9502, -0.6321, -0.3935, 0.5000, 2.0000])9print(x.grad) # tensor([0.0498, 0.3679, 0.6065, 1.0000, 1.0000])1011class MyELU(torch.autograd.Function):12 @staticmethod13 def forward(ctx, x, alpha=1.0):14 y = torch.where(x > 0, x, alpha * torch.expm1(x)) # expm1 is accurate near 015 ctx.save_for_backward(x, y)16 ctx.alpha = alpha17 return y18 @staticmethod19 def backward(ctx, g):20 x, y = ctx.saved_tensors21 return g * torch.where(x > 0, torch.ones_like(y), y + ctx.alpha), NoneUse when
- +Mean activation closer to zero than ReLU (less bias shift)
- +Identity on the right: no saturation for positive inputs
- +Gradient never exactly zero; smooth at 0 when α = 1
- +Bounded below: a noise-robust off state
Avoid when
- −An exp per negative input: slower than ReLU
- −Still saturates (softly) for very negative inputs
- −He init over-amplifies it slightly; needs a matched gain or normalization
- −Smoothness only at α = 1; not sparse (no exact zeros)
Check yourself
For which value of α is ELU continuously differentiable at 0?
Check yourself
An ELU unit (α = 1) output y = −0.8 during the forward pass. What is its local derivative, without recomputing any exponential?
Remember
- ✓ for , for ; derivative 1 or .
- ✓Negative outputs pull the mean toward zero: for standard normal input, vs ReLU's .
- ✓Smooth () only at ; saturates softly at ; costs an exponential.
- ✓Clevert et al. (2015) argued a near-zero mean reduces bias shift and brings the gradient closer to the natural gradient.
- ✓Scaling ELU with by gives SELU.