Definition
Softplus is the activation . It is a smooth approximation of that is strictly positive everywhere, and its derivative is the logistic sigmoid .
Softplus is the integral of the sigmoid: . The sigmoid is a smooth step and ReLU's slope is a hard step, so softplus relates to ReLU the way the sigmoid relates to the step function.
Dugas, Bengio, Bélisle, Nadeau & Garcia introduced it as a neural-network nonlinearity in 2001, for option pricing, where a convex, positive, smooth function was the right prior. Glorot, Bordes & Bengio (2011) compared it with ReLU in deep networks and found the rectifier performed as well or better despite its hard zero and kink. ReLU won the hidden-layer role. Softplus kept another job: turning an unconstrained number into a positive one.
Formula and derivative
- logistic sigmoid; the slope is in and never exactly 0
- always positive: softplus is strictly convex, with peak curvature at
Its distance from ReLU is . That gap is symmetric, largest at the origin () and decays like : at and at .
Sharpness β, and the stable form
A sharpness parameter trades smoothness for closeness to ReLU:
- sharpness; gives ReLU, and the slope is
- the largest gap to ReLU, at ; e.g. for
Written literally, breaks in floating point at both ends. For large , overflows to inf. For very negative , is smaller than half the spacing of floats near 1, so rounds to exactly 1 and the log returns 0. The fix is an exact rewrite:
- always in , so it cannot overflow
- computed accurately for tiny , where
In float32 the naive form returns inf for ( of the largest float32) and returns exactly 0 for (). Between about and it returns a number with a large relative error. In float64 the limits move to and but do not go away. The stable form is accurate to float32 rounding across the whole range.
At a glance
- Range
- : strictly positive, unbounded above
- Monotonic
- Yes, strictly increasing (slope σ(x) > 0)
- Smooth
- Yes, infinitely differentiable and strictly convex
- Zero-centred
- No. For the output has mean and std
- Saturation
- The slope saturates to 0 as (like the sigmoid) but never reaches it; no saturation above
- Cost
- An exp and a log1p per element: far more than ReLU's compare
Worked example
| x | softplus(x) | σ(x) = slope | gap to ReLU | naive float32 |
|---|---|---|---|---|
| −20 | 2.0612e−9 | 2.0612e−9 | 2.0612e−9 | 0 (wrong) |
| −5 | 0.0067 | 0.0067 | 0.0067 | 0.0067 |
| −1 | 0.3133 | 0.2689 | 0.3133 | 0.3133 |
| 0 | 0.6931 | 0.5000 | 0.6931 | 0.6931 |
| 1 | 1.3133 | 0.7311 | 0.3133 | 1.3133 |
| 5 | 5.0067 | 0.9933 | 0.0067 | 5.0067 |
| 20 | 20.0000 | 1.0000 | 2.0612e−9 | 20.0000 |
| 100 | 100.0000 | 1.0000 | ≈ 3.7e−44 | inf (wrong) |
For very negative , , which is why the second and third columns agree at . For large , the output is and the slope is 1, exactly ReLU.
Where it is used today
Where it's used
Positive scales and variances
Gaussian output heads, mixture density networks and probabilistic regressors predict . It grows linearly rather than exponentially, so a large raw value does not explode the scale the way can. VAEs often predict instead; softplus is the common alternative.
Mamba's step size Δ
Selective state-space models (Mamba, Gu & Dao 2023) compute their input-dependent discretisation step as , which keeps it positive. The bias is initialised through the inverse softplus so that starts in a chosen range.
Constrained parameters
Any learned quantity that must be positive (a temperature, a length-scale, a precision) can be stored as an unconstrained and used as . To start at value , initialise , e.g. for .
Inside losses
Binary cross-entropy with logits is , and F.binary_cross_entropy_with_logits computes it with the same stable trick.
Effect on training
As a hidden activation, softplus has no dead units (the slope is never 0), but it saturates on the negative side like a sigmoid. Its outputs are all positive with mean for unit-variance inputs, which biases the next layer's pre-activations. It is also convex, so a softplus unit cannot produce the small negative dip of GELU or Swish. These are the reasons ReLU-family and GELU-family activations displaced it. As an output transform its gradient is well behaved: close to 1 when the predicted value is large and decaying smoothly toward 0 as the value approaches its floor.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([-2., -1., 0., 1., 2.])4print(F.softplus(x)) # [0.1269, 0.3133, 0.6931, 1.3133, 2.1269]56big = torch.tensor([-20., 100.]) # float327print(torch.log(1 + torch.exp(big))) # [0., inf] <- naive form fails at both ends8print(F.softplus(big)) # [2.0612e-09, 100.]910def softplus(x, beta=1.0): # from scratch, stable11 bx = beta * x12 return (bx.clamp(min=0) + torch.log1p(torch.exp(-bx.abs()))) / beta1314sp = nn.Softplus(beta=1.0, threshold=20.0) # returns x itself where beta*x > 201516# positive output head, initialised so that sigma starts at 1.017head = nn.Linear(64, 1)18nn.init.zeros_(head.weight); nn.init.constant_(head.bias, torch.log(torch.expm1(torch.tensor(1.0))).item()) # 0.541319sigma = F.softplus(head(torch.randn(8, 64))) + 1e-6 # all ≈ 1.0Use when
- +Smooth, convex and strictly positive: the natural transform for scales, variances and step sizes.
- +Derivative is the sigmoid, never exactly 0, so there are no dead units.
- +β tunes it continuously toward ReLU, with an exact error bound ln 2/β.
Avoid when
- −As a hidden activation it is beaten or matched by ReLU at a fraction of the cost.
- −Not zero-centred (mean 0.806 for unit Gaussian inputs) and saturating on the negative side.
- −The literal formula overflows above 88.7 and underflows below −16.6 in float32.
Check yourself
What is the largest gap between softplus_β and ReLU, and where does it occur?
Check yourself
In float32, the naive log(1 + exp(x)) returns 0 for x = −20. Why?
Remember
- ✓ (Dugas et al. 2001); , .
- ✓Gap to ReLU is , at most ; with sharpness it is at most .
- ✓Compute it as ; the literal form fails above 88.72 and below −16.64 in float32.
- ✓Its main modern role is positivity: Gaussian scales, Mamba's Δ, learned temperatures; initialise through ln(eʸ − 1).
- ✓As a hidden activation ReLU matches or beats it (Glorot et al. 2011).