Definition
Softsign is : an odd, S-shaped function with range and slope 1 at the origin, like tanh, whose tails approach like instead of exponentially fast.
The function appeared early as the "Elliott" activation (Elliott, 1993), a sigmoid substitute without exponentials, and MATLAB still ships it as elliotsig. The name softsign comes from deep learning. Glorot and Bengio's 2010 study of why deep networks were hard to train compared it with the sigmoid and tanh, crediting it to Bergstra et al. (2009).
They reported that softsign networks were more robust to the initialization procedure than tanh networks, which they attributed to its gentler nonlinearity. After training, softsign activations tended to sit around the "knees" of the curve rather than at the asymptotes. It is available in Keras, TensorFlow and PyTorch but is rare in modern architectures: ReLU-family activations made saturation-tolerant squashing functions mostly unnecessary.
Formula and derivative
- absolute value: the only non-polynomial piece, and the reason the function is not twice differentiable at 0
- slope 1 at x = 0, decaying like 1/x², not exponentially
- closed-form inverse: reaching 0.9 needs x = 9 (tanh needs 1.47)
- softsign at x = 10 is 0.909: still 9% from its asymptote
- tanh at is within of 1
Properties
- Range
- Monotonic
- strictly increasing
- Smooth
- only: jumps from to at
- Zero-centred
- yes: odd, f(0) = 0
- Saturating
- yes, but polynomially: (tanh: )
- Max slope
- 1 at x = 0, like tanh
- Cost
- an abs, an add and a divide; no exponential
Polynomial versus exponential saturation
The two derivatives cross at . Inside that, tanh is steeper: at , against softsign's . Outside it, softsign is steeper, and the gap grows exponentially. A unit passes less than 5% of the gradient once for tanh, but only once for softsign. At the ratio is to , a million to one.
In float64, rounds to exactly 1, so the derivative computed from it is exactly 0. In float32 this already happens around . Softsign's derivative at 20 is still .
Worked example
| x | softsign(x) | softsign′(x) | tanh(x) | tanh′(x) |
|---|---|---|---|---|
| 0 | 0.0000 | 1.0000 | 0.0000 | 1.0000 |
| 0.5 | 0.3333 | 0.4444 | 0.4621 | 0.7864 |
| 1 | 0.5000 | 0.2500 | 0.7616 | 0.4200 |
| 2 | 0.6667 | 0.1111 | 0.9640 | 0.0707 |
| 3 | 0.7500 | 0.0625 | 0.9951 | 0.0099 |
| 5 | 0.8333 | 0.0278 | 0.9999 | |
| 10 | 0.9091 | 0.0083 | 1.0000 |
Forward and backward at : and , using only an abs, an add, a divide and a square. tanh at the same point passes of the gradient, about 6 times less.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([-10.0, -1.0, 0.0, 3.0, 10.0], requires_grad=True)45act = nn.Softsign() # module6y = F.softsign(x) # functional78def softsign(x): # from scratch: no exponential9 return x / (1 + x.abs())1011def dsoftsign(x):12 return 1 / (1 + x.abs()) ** 2 # 1 at 0, decays like 1/x²1314softsign(x).sum().backward()15print(x.grad) # tensor([0.0083, 0.2500, 1.0000, 0.0625, 0.0083])1617# Drop-in replacement for tanh in an MLP18mlp = nn.Sequential(nn.Linear(784, 512), nn.Softsign(), nn.Linear(512, 10))Use when
- +Zero-centred, bounded, slope 1 at 0: tanh's good properties
- +Polynomial tails: saturated units still pass some gradient
- +No exponential: cheap to compute
- +Closed-form inverse
Avoid when
- −Flatter than tanh for , so the signal shrinks faster near 0
- −Still saturating: gradients vanish with depth
- −Only : the curvature jumps at 0
- −Little evidence of gains over tanh or ReLU in modern architectures, so it is rarely used
Check yourself
At x = 5, roughly how much more gradient does softsign pass than tanh?
Check yourself
For which inputs does tanh have the LARGER derivative?
Remember
- ✓, , slope 1 at 0; inverse .
- ✓Polynomial tails (, ) versus tanh's exponential ones: at , versus .
- ✓Flatter than tanh for , steeper beyond: it saturates later and more gently.
- ✓Glorot & Bengio (2010) found softsign nets more robust to initialization than tanh nets; also known as the Elliott function (1993).
- ✓Cheap (no exp) and available in PyTorch and Keras, but rarely used today: ReLU-family activations sidestep saturation altogether.