# Softsign

> x/(1 + |x|) has tanh's shape and range but approaches ±1 polynomially rather than exponentially, so it saturates more gently and needs no exp. It works, but it is rarely used.

Source: https://www.mlfoundry.in/learn/softsign  
Section: Deep Learning › Activation Functions  
Level: advanced · 7 min read · updated 2026-10-11

**Softsign:** is $\operatorname{softsign}(x)=\dfrac{x}{1+|x|}$: an odd, S-shaped function with range $(-1,1)$ and slope 1 at the origin, like tanh, whose tails approach $\pm1$ like $1/|x|$ instead of exponentially fast.

The function appeared early as the "Elliott" activation (Elliott, 1993), a sigmoid substitute without exponentials, and MATLAB still ships it as `elliotsig`. The name **softsign** comes from deep learning. Glorot and Bengio's 2010 study of why deep networks were hard to train compared it with the sigmoid and tanh, crediting it to Bergstra et al. (2009).

They reported that softsign networks were more robust to the initialization procedure than tanh networks, which they attributed to its gentler nonlinearity. After training, softsign activations tended to sit around the "knees" of the curve rather than at the asymptotes. It is available in Keras, TensorFlow and PyTorch but is rare in modern architectures: ReLU-family activations made saturation-tolerant squashing functions mostly unnecessary.

## Formula and derivative

$$
\operatorname{softsign}(x)=\frac{x}{1+|x|},\qquad \operatorname{softsign}'(x)=\frac{1}{(1+|x|)^2},\qquad \operatorname{softsign}^{-1}(s)=\frac{s}{1-|s|}
$$

- $|x|$: absolute value: the only non-polynomial piece, and the reason the function is not twice differentiable at 0
- $(1+|x|)^{-2}$: slope 1 at x = 0, decaying like 1/x², not exponentially
- $s/(1-|s|)$: closed-form inverse: reaching 0.9 needs x = 9 (tanh needs 1.47)

$$
1-\operatorname{softsign}(x)=\frac{1}{1+x}\quad\text{vs}\quad 1-\tanh(x)\approx2e^{-2x},\qquad \operatorname{softsign}'(x)\sim x^{-2}\quad\text{vs}\quad\tanh'(x)\approx4e^{-2x}\qquad(x\to\infty)
$$

- $\text{polynomial tail}$: softsign at x = 10 is 0.909: still 9% from its asymptote
- $\text{exponential tail}$: tanh at $x=10$ is within $4\times10^{-9}$ of 1

**Properties**

- **Range:** $(-1,1)$
- **Monotonic:** strictly increasing
- **Smooth:** $C^1$ only: $f''$ jumps from $+2$ to $-2$ at $x=0$
- **Zero-centred:** yes: odd, f(0) = 0
- **Saturating:** yes, but polynomially: $f'(5)=0.028$ (tanh: $1.8\times10^{-4}$)
- **Max slope:** 1 at x = 0, like tanh
- **Cost:** an abs, an add and a divide; no exponential

> Figure: Softsign against tanh: identical at 0 (value 0, slope 1). Between, softsign bends earlier, so it is below tanh. In the tails it approaches ±1 much more slowly, and its derivative stays visibly above zero long after tanh's has flattened.

## Polynomial versus exponential saturation

The two derivatives cross at $|x|=1.616$. Inside that, tanh is **steeper**: at $x=1$, $\tanh'=0.42$ against softsign's $0.25$. Outside it, softsign is steeper, and the gap grows exponentially. A unit passes less than 5% of the gradient once $|x|>2.18$ for tanh, but only once $|x|>3.47$ for softsign. At $x=10$ the ratio is $0.0083$ to $8.2\times10^{-9}$, a million to one.

In float64, $\tanh(20)$ rounds to exactly 1, so the derivative computed from it is exactly 0. In float32 this already happens around $x\approx9$. Softsign's derivative at 20 is still $0.0023$.

> Figure: Top: both derivatives on a log scale. At the default probe x = 5, tanh′ = 1.8e−4 and softsign′ = 0.028. Bottom: activation histograms in each of 5 layers of 64 units at initialization (W ~ U(±g/√n), 64 Gaussian inputs). At g = 1, the 'standard' initialization Glorot & Bengio studied, both shrink toward 0 with depth, softsign faster, because it is less steep near 0. Press g = 6: tanh piles up at ±1, with 46–52% of units flat (f′ < 0.05) in every layer. Softsign spreads across the curve, with 31% flat in layer 1 falling to 7% by layer 5.

## Worked example

softsign and tanh side by side (computed in Node)

| x | softsign(x) | softsign′(x) | tanh(x) | tanh′(x) |
| --- | --- | --- | --- | --- |
| 0 | 0.0000 | 1.0000 | 0.0000 | 1.0000 |
| 0.5 | 0.3333 | 0.4444 | 0.4621 | 0.7864 |
| 1 | 0.5000 | 0.2500 | 0.7616 | 0.4200 |
| 2 | 0.6667 | 0.1111 | 0.9640 | 0.0707 |
| 3 | 0.7500 | 0.0625 | 0.9951 | 0.0099 |
| 5 | 0.8333 | 0.0278 | 0.9999 | $1.8\times10^{-4}$ |
| 10 | 0.9091 | 0.0083 | 1.0000 | $8.2\times10^{-9}$ |

Forward and backward at $x=3$: $f(3)=3/4=0.75$ and $f'(3)=1/16=0.0625$, using only an abs, an add, a divide and a square. tanh at the same point passes $0.0099$ of the gradient, about 6 times less.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-10.0, -1.0, 0.0, 3.0, 10.0], requires_grad=True)

act = nn.Softsign()               # module
y = F.softsign(x)                 # functional

def softsign(x):                  # from scratch: no exponential
    return x / (1 + x.abs())

def dsoftsign(x):
    return 1 / (1 + x.abs()) ** 2     # 1 at 0, decays like 1/x²

softsign(x).sum().backward()
print(x.grad)                     # tensor([0.0083, 0.2500, 1.0000, 0.0625, 0.0083])

# Drop-in replacement for tanh in an MLP
mlp = nn.Sequential(nn.Linear(784, 512), nn.Softsign(), nn.Linear(512, 10))
```

Softsign in PyTorch: module, functional and from scratch

> **Gentle saturation is still saturation:** Polynomial tails make saturation less abrupt, not absent: $f'(10)=0.0083$ is still a strong attenuation per layer, and it compounds with depth. Near the origin softsign is **flatter** than tanh, so at small initial scales its activations shrink faster, as the figure shows at $g=1$. It needs the same care with initialization as tanh.

> **Why heavy tails help optimization:** With tanh, a unit pushed to $|x|=5$ is effectively frozen ($\tanh'\approx1.8\times10^{-4}$), and getting it back takes a gradient thousands of times larger than normal. A softsign unit at the same point still passes about 3% of its gradient, so it can be pulled back. That matches Glorot and Bengio's observation that softsign was less sensitive to initialization.

**Strengths**

- Zero-centred, bounded, slope 1 at 0: tanh's good properties
- Polynomial tails: saturated units still pass some gradient
- No exponential: cheap to compute
- Closed-form inverse

**Limitations**

- Flatter than tanh for $|x|<1.6$, so the signal shrinks faster near 0
- Still saturating: gradients vanish with depth
- Only $C^1$: the curvature jumps at 0
- Little evidence of gains over tanh or ReLU in modern architectures, so it is rarely used

**Question:** At x = 5, roughly how much more gradient does softsign pass than tanh?

- A. About the same
- B. About 2×
- C. About 150×
- D. About 10⁶×

**Answer:** C. About 150× — $\operatorname{softsign}'(5)=1/36=0.0278$ and $\tanh'(5)=1-\tanh^2(5)=1.82\times10^{-4}$. The ratio is about 153. At $x=10$ it grows to about $10^6$, because one tail is polynomial and the other exponential.

**Question:** For which inputs does tanh have the LARGER derivative?

- A. Never: softsign is always steeper
- B. For |x| < 1.616
- C. For |x| > 1.616
- D. Only at x = 0

**Answer:** B. For |x| < 1.616 — Both have slope 1 at 0, but $\operatorname{softsign}'(1)=0.25<\tanh'(1)=0.42$. The curves cross at $|x|\approx1.616$, and beyond that softsign's $1/(1+|x|)^2$ decays far more slowly than $\operatorname{sech}^2 x$.

**Key points**

- $\operatorname{softsign}(x)=x/(1+|x|)\in(-1,1)$, $f'(x)=1/(1+|x|)^2$, slope 1 at 0; inverse $s/(1-|s|)$.
- Polynomial tails ($1-f\sim1/x$, $f'\sim1/x^2$) versus tanh's exponential ones: at $x=10$, $0.0083$ versus $8.2\times10^{-9}$.
- Flatter than tanh for $|x|<1.616$, steeper beyond: it saturates later and more gently.
- Glorot & Bengio (2010) found softsign nets more robust to initialization than tanh nets; also known as the Elliott function (1993).
- Cheap (no exp) and available in PyTorch and Keras, but rarely used today: ReLU-family activations sidestep saturation altogether.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
