Softsign

x/(1 + |x|) has tanh's shape and range but approaches ±1 polynomially rather than exponentially, so it saturates more gently and needs no exp. It works, but it is rarely used.

7 min read449 wordsadvanced0 viewsupdated 2026-10-11

Definition

Softsign is softsign⁡(x)=x1+∣x∣\operatorname{softsign}(x)=\dfrac{x}{1+|x|}: an odd, S-shaped function with range (−1,1)(-1,1) and slope 1 at the origin, like tanh, whose tails approach ±1\pm1 like 1/∣x∣1/|x| instead of exponentially fast.

The function appeared early as the "Elliott" activation (Elliott, 1993), a sigmoid substitute without exponentials, and MATLAB still ships it as elliotsig. The name softsign comes from deep learning. Glorot and Bengio's 2010 study of why deep networks were hard to train compared it with the sigmoid and tanh, crediting it to Bergstra et al. (2009).

They reported that softsign networks were more robust to the initialization procedure than tanh networks, which they attributed to its gentler nonlinearity. After training, softsign activations tended to sit around the "knees" of the curve rather than at the asymptotes. It is available in Keras, TensorFlow and PyTorch but is rare in modern architectures: ReLU-family activations made saturation-tolerant squashing functions mostly unnecessary.

Formula and derivative

softsign⁡(x)=x1+∣x∣,softsign⁡′(x)=1(1+∣x∣)2,softsign⁡−1(s)=s1−∣s∣\operatorname{softsign}(x)=\frac{x}{1+|x|},\qquad \operatorname{softsign}'(x)=\frac{1}{(1+|x|)^2},\qquad \operatorname{softsign}^{-1}(s)=\frac{s}{1-|s|}
∣x∣|x|
absolute value: the only non-polynomial piece, and the reason the function is not twice differentiable at 0
(1+∣x∣)−2(1+|x|)^{-2}
slope 1 at x = 0, decaying like 1/x², not exponentially
s/(1−∣s∣)s/(1-|s|)
closed-form inverse: reaching 0.9 needs x = 9 (tanh needs 1.47)
1−softsign⁡(x)=11+xvs1−tanh⁡(x)≈2e−2x,softsign⁡′(x)∼x−2vstanh⁡′(x)≈4e−2x(x→∞)1-\operatorname{softsign}(x)=\frac{1}{1+x}\quad\text{vs}\quad 1-\tanh(x)\approx2e^{-2x},\qquad \operatorname{softsign}'(x)\sim x^{-2}\quad\text{vs}\quad\tanh'(x)\approx4e^{-2x}\qquad(x\to\infty)
polynomial tail\text{polynomial tail}
softsign at x = 10 is 0.909: still 9% from its asymptote
exponential tail\text{exponential tail}
tanh at x=10x=10 is within 4×10−94\times10^{-9} of 1

Properties

Range
(−1,1)(-1,1)
Monotonic
strictly increasing
Smooth
C1C^1 only: f′′f'' jumps from +2+2 to −2-2 at x=0x=0
Zero-centred
yes: odd, f(0) = 0
Saturating
yes, but polynomially: f′(5)=0.028f'(5)=0.028 (tanh: 1.8×10−41.8\times10^{-4})
Max slope
1 at x = 0, like tanh
Cost
an abs, an add and a divide; no exponential
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 0.500 · f′(x) = 0.250−1.0−0.50.00.51.0−4−2024x →SoftsignSoftsign f′Tanh
activation
compare
range (−1, 1)monotonic yessmooth nozero-centred yes
C¹ but not C² (f″ jumps at 0). Approaches ±1 polynomially, so it saturates more gently than tanh.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]−0.800.000.80output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.000Var[f(x)] 0.183E[f′(x)] 0.413f′ = 0 for 0.0% of inputsE[f′]¹⁰ 1.4e−4
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.Softsign against tanh: identical at 0 (value 0, slope 1). Between, softsign bends earlier, so it is below tanh. In the tails it approaches ±1 much more slowly, and its derivative stays visibly above zero long after tanh's has flattened.

Polynomial versus exponential saturation

The two derivatives cross at ∣x∣=1.616|x|=1.616. Inside that, tanh is steeper: at x=1x=1, tanh⁡′=0.42\tanh'=0.42 against softsign's 0.250.25. Outside it, softsign is steeper, and the gap grows exponentially. A unit passes less than 5% of the gradient once ∣x∣>2.18|x|>2.18 for tanh, but only once ∣x∣>3.47|x|>3.47 for softsign. At x=10x=10 the ratio is 0.00830.0083 to 8.2×10−98.2\times10^{-9}, a million to one.

In float64, tanh⁡(20)\tanh(20) rounds to exactly 1, so the derivative computed from it is exactly 0. In float32 this already happens around x≈9x\approx9. Softsign's derivative at 20 is still 0.00230.0023.

f′(x) on a log scale · tanh′ = 1 − tanh² ≈ 4e^(−2x) · softsign′ = 1/(1 + x)²11e-21e-41e-61e-81e-10f′ = 0.0505101520x = 5.0: tanh 0.99991, tanh′ 1.82e-4 · softsign 0.8333, ′ 0.0278activations at init · 5 layers × 64 units, batch 64 · W ~ U(±g/√n)tanhE f′ 0.79flat 0%L1E f′ 0.94flat 0%L2E f′ 0.98flat 0%L3E f′ 0.99flat 0%L4E f′ 1.00flat 0%L5−101softsignE f′ 0.55flat 0%E f′ 0.78flat 0%E f′ 0.88flat 0%E f′ 0.94flat 0%E f′ 0.96flat 0%−101flat = units with f′(z) < 0.05, passing almost no gradient · E f′ = mean slope
presets
fig.Top: both derivatives on a log scale. At the default probe x = 5, tanh′ = 1.8e−4 and softsign′ = 0.028. Bottom: activation histograms in each of 5 layers of 64 units at initialization (W ~ U(±g/√n), 64 Gaussian inputs). At g = 1, the 'standard' initialization Glorot & Bengio studied, both shrink toward 0 with depth, softsign faster, because it is less steep near 0. Press g = 6: tanh piles up at ±1, with 46–52% of units flat (f′ < 0.05) in every layer. Softsign spreads across the curve, with 31% flat in layer 1 falling to 7% by layer 5.

Worked example

xsoftsign(x)softsign′(x)tanh(x)tanh′(x)
00.00001.00000.00001.0000
0.50.33330.44440.46210.7864
10.50000.25000.76160.4200
20.66670.11110.96400.0707
30.75000.06250.99510.0099
50.83330.02780.99991.8×10−41.8\times10^{-4}
100.90910.00831.00008.2×10−98.2\times10^{-9}
softsign and tanh side by side (computed in Node)

Forward and backward at x=3x=3: f(3)=3/4=0.75f(3)=3/4=0.75 and f′(3)=1/16=0.0625f'(3)=1/16=0.0625, using only an abs, an add, a divide and a square. tanh at the same point passes 0.00990.0099 of the gradient, about 6 times less.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-10.0, -1.0, 0.0, 3.0, 10.0], requires_grad=True)
4
5act = nn.Softsign() # module
6y = F.softsign(x) # functional
7
8def softsign(x): # from scratch: no exponential
9 return x / (1 + x.abs())
10
11def dsoftsign(x):
12 return 1 / (1 + x.abs()) ** 2 # 1 at 0, decays like 1/x²
13
14softsign(x).sum().backward()
15print(x.grad) # tensor([0.0083, 0.2500, 1.0000, 0.0625, 0.0083])
16
17# Drop-in replacement for tanh in an MLP
18mlp = nn.Sequential(nn.Linear(784, 512), nn.Softsign(), nn.Linear(512, 10))
Softsign in PyTorch: module, functional and from scratch

Use when

  • +Zero-centred, bounded, slope 1 at 0: tanh's good properties
  • +Polynomial tails: saturated units still pass some gradient
  • +No exponential: cheap to compute
  • +Closed-form inverse

Avoid when

  • −Flatter than tanh for ∣x∣<1.6|x|<1.6, so the signal shrinks faster near 0
  • −Still saturating: gradients vanish with depth
  • −Only C1C^1: the curvature jumps at 0
  • −Little evidence of gains over tanh or ReLU in modern architectures, so it is rarely used

Check yourself

At x = 5, roughly how much more gradient does softsign pass than tanh?

Check yourself

For which inputs does tanh have the LARGER derivative?

Remember

  • ✓softsign⁡(x)=x/(1+∣x∣)∈(−1,1)\operatorname{softsign}(x)=x/(1+|x|)\in(-1,1), f′(x)=1/(1+∣x∣)2f'(x)=1/(1+|x|)^2, slope 1 at 0; inverse s/(1−∣s∣)s/(1-|s|).
  • ✓Polynomial tails (1−f∼1/x1-f\sim1/x, f′∼1/x2f'\sim1/x^2) versus tanh's exponential ones: at x=10x=10, 0.00830.0083 versus 8.2×10−98.2\times10^{-9}.
  • ✓Flatter than tanh for ∣x∣<1.616|x|<1.616, steeper beyond: it saturates later and more gently.
  • ✓Glorot & Bengio (2010) found softsign nets more robust to initialization than tanh nets; also known as the Elliott function (1993).
  • ✓Cheap (no exp) and available in PyTorch and Keras, but rarely used today: ReLU-family activations sidestep saturation altogether.
← back to Deep Learning roadmap