Softplus

log(1 + eˣ): a smooth, always-positive ReLU whose derivative is the sigmoid. It is rarely a hidden activation today, but it is the standard way to keep an output positive.

7 min read579 wordscore0 viewsupdated 2026-10-11

Definition

Softplus is the activation softplus(x)=ln⁡(1+ex)\mathrm{softplus}(x)=\ln(1+e^{x}). It is a smooth approximation of max⁡(0,x)\max(0,x) that is strictly positive everywhere, and its derivative is the logistic sigmoid σ(x)\sigma(x).

Softplus is the integral of the sigmoid: ∫−∞xσ(t) dt=ln⁡(1+ex)\int_{-\infty}^{x}\sigma(t)\,dt=\ln(1+e^x). The sigmoid is a smooth step and ReLU's slope is a hard step, so softplus relates to ReLU the way the sigmoid relates to the step function.

Dugas, Bengio, Bélisle, Nadeau & Garcia introduced it as a neural-network nonlinearity in 2001, for option pricing, where a convex, positive, smooth function was the right prior. Glorot, Bordes & Bengio (2011) compared it with ReLU in deep networks and found the rectifier performed as well or better despite its hard zero and kink. ReLU won the hidden-layer role. Softplus kept another job: turning an unconstrained number into a positive one.

Formula and derivative

softplus(x)=ln⁡(1+ex),softplus′(x)=ex1+ex=σ(x),softplus′′(x)=σ(x)(1−σ(x))\mathrm{softplus}(x)=\ln(1+e^{x}),\qquad \mathrm{softplus}'(x)=\frac{e^x}{1+e^x}=\sigma(x),\qquad \mathrm{softplus}''(x)=\sigma(x)\big(1-\sigma(x)\big)
σ(x)\sigma(x)
logistic sigmoid; the slope is in (0,1)(0,1) and never exactly 0
σ(1−σ)\sigma(1-\sigma)
always positive: softplus is strictly convex, with peak curvature 0.250.25 at x=0x=0

Its distance from ReLU is softplus(x)−max⁡(0,x)=ln⁡(1+e−∣x∣)\mathrm{softplus}(x)-\max(0,x)=\ln(1+e^{-|x|}). That gap is symmetric, largest at the origin (ln⁡2=0.6931\ln2=0.6931) and decays like e−∣x∣e^{-|x|}: 0.12690.1269 at ∣x∣=2|x|=2 and 0.00670.0067 at ∣x∣=5|x|=5.

Sharpness β, and the stable form

A sharpness parameter β\beta trades smoothness for closeness to ReLU:

softplusβ(x)=1βln⁡ ⁣(1+eβx),max⁡x∣softplusβ(x)−max⁡(0,x)∣=ln⁡2β\mathrm{softplus}_\beta(x)=\frac1\beta\ln\!\big(1+e^{\beta x}\big),\qquad \max_x\big|\mathrm{softplus}_\beta(x)-\max(0,x)\big|=\frac{\ln 2}{\beta}
β\beta
sharpness; β→∞\beta\to\infty gives ReLU, and the slope is σ(βx)\sigma(\beta x)
ln⁡2/β\ln2/\beta
the largest gap to ReLU, at x=0x=0; e.g. 0.06930.0693 for β=10\beta=10

Written literally, ln⁡(1+ex)\ln(1+e^x) breaks in floating point at both ends. For large xx, exe^x overflows to inf. For very negative xx, exe^x is smaller than half the spacing of floats near 1, so 1+ex1+e^x rounds to exactly 1 and the log returns 0. The fix is an exact rewrite:

ln⁡(1+ex)=max⁡(x,0)+log1p⁡ ⁣(e−∣x∣)\ln(1+e^{x})=\max(x,0)+\operatorname{log1p}\!\big(e^{-|x|}\big)
e−∣x∣e^{-|x|}
always in (0,1](0,1], so it cannot overflow
log1p⁡(u)\operatorname{log1p}(u)
ln⁡(1+u)\ln(1+u) computed accurately for tiny uu, where ln⁡(1+u)≈u\ln(1+u)\approx u

In float32 the naive form returns inf for x>88.72x>88.72 (=ln⁡=\ln of the largest float32) and returns exactly 0 for x<−16.64x<-16.64 (ex<2−24e^x<2^{-24}). Between about −16-16 and −5-5 it returns a number with a large relative error. In float64 the limits move to 709.78709.78 and −36.74-36.74 but do not go away. The stable form is accurate to float32 rounding across the whole range.

(1/β)·log(1 + e^(βx)) vs ReLU, β = 1.000123-3-2-10123gap at 0 = log 2/β = 0.6931slope = σ(βx) → step00.51-303float32 relative error vs exact, x ∈ [-80, 120] (log scale)inf1e-91e-61e-3100%-80-40-16.604088.7120— naive log(1+eˣ) — stable max(x,0)+log1p(e^−|x|) ■ 1+eˣ rounds to 1 ■ eˣ overflows
jump to
float32 eˣ 2.0612e-9float32 1 + eˣ 1naive 0 (eˣ swallowed by the 1)stable 2.0612e-9 (rel. err 3.4861e-8)exact 2.0612e-9max gap to ReLU 0.6931 = log 2/β
fig.Top: drag β. The shaded gap to ReLU shrinks, and its maximum (at x = 0) is always log 2/β; on the right the slope σ(βx) sharpens into ReLU's step. Bottom: the relative error of each implementation, computed live with every operation rounded to float32. The naive form (rose) is 100% wrong left of −16.6 and infinite right of 88.7; the stable form (green) stays near 1e-7. Move the probe or jump to −20 and 89 to see the intermediate values.
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 1.313 · f′(x) = 0.731−1012345−4−2024x →Softplus (β=1)Softplus f′ReLUSigmoid
activation
compare
range (0, ∞)monotonic yessmooth yeszero-centred no
A smooth ReLU; its derivative is the sigmoid. β → ∞ approaches ReLU; f(0) = ln 2 / β.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]0.022.024.01output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.806Var[f(x)] 0.272E[f′(x)] 0.500f′ = 0 for 0.0% of inputsE[f′]¹⁰ 9.8e−4
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.Softplus against ReLU, with the sigmoid for reference: the derivative panel shows softplus′ lying exactly on the sigmoid curve.

At a glance

Range
(0,∞)(0,\infty): strictly positive, unbounded above
Monotonic
Yes, strictly increasing (slope σ(x) > 0)
Smooth
Yes, infinitely differentiable and strictly convex
Zero-centred
No. For x∼N(0,1)x\sim\mathcal N(0,1) the output has mean 0.8060.806 and std 0.5210.521
Saturation
The slope saturates to 0 as x→−∞x\to-\infty (like the sigmoid) but never reaches it; no saturation above
Cost
An exp and a log1p per element: far more than ReLU's compare

Worked example

xsoftplus(x)σ(x) = slopegap to ReLUnaive float32
−202.0612e−92.0612e−92.0612e−90 (wrong)
−50.00670.00670.00670.0067
−10.31330.26890.31330.3133
00.69310.50000.69310.6931
11.31330.73110.31331.3133
55.00670.99330.00675.0067
2020.00001.00002.0612e−920.0000
100100.00001.0000≈ 3.7e−44inf (wrong)
Softplus, its slope and its gap to ReLU (highlighted: the largest gap; the last column is the naive formula in float32)

For very negative xx, softplus(x)≈ex≈σ(x)\mathrm{softplus}(x)\approx e^x\approx\sigma(x), which is why the second and third columns agree at −20-20. For large xx, the output is xx and the slope is 1, exactly ReLU.

Where it is used today

Where it's used

Positive scales and variances

Gaussian output heads, mixture density networks and probabilistic regressors predict σ=softplus(a)+ε\sigma=\mathrm{softplus}(a)+\varepsilon. It grows linearly rather than exponentially, so a large raw value does not explode the scale the way eae^a can. VAEs often predict log⁡σ2\log\sigma^2 instead; softplus is the common alternative.

Mamba's step size Δ

Selective state-space models (Mamba, Gu & Dao 2023) compute their input-dependent discretisation step as Δ=softplus(linear(x)+b)\Delta=\mathrm{softplus}(\text{linear}(x)+b), which keeps it positive. The bias is initialised through the inverse softplus so that Δ\Delta starts in a chosen range.

Constrained parameters

Any learned quantity that must be positive (a temperature, a length-scale, a precision) can be stored as an unconstrained aa and used as softplus(a)\mathrm{softplus}(a). To start at value yy, initialise a=ln⁡(ey−1)a=\ln(e^{y}-1), e.g. 0.54130.5413 for y=1y=1.

Inside losses

Binary cross-entropy with logits is softplus(z)−yz\mathrm{softplus}(z)-yz, and F.binary_cross_entropy_with_logits computes it with the same stable trick.

Effect on training

As a hidden activation, softplus has no dead units (the slope is never 0), but it saturates on the negative side like a sigmoid. Its outputs are all positive with mean 0.8060.806 for unit-variance inputs, which biases the next layer's pre-activations. It is also convex, so a softplus unit cannot produce the small negative dip of GELU or Swish. These are the reasons ReLU-family and GELU-family activations displaced it. As an output transform its gradient σ(a)\sigma(a) is well behaved: close to 1 when the predicted value is large and decaying smoothly toward 0 as the value approaches its floor.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-2., -1., 0., 1., 2.])
4print(F.softplus(x)) # [0.1269, 0.3133, 0.6931, 1.3133, 2.1269]
5
6big = torch.tensor([-20., 100.]) # float32
7print(torch.log(1 + torch.exp(big))) # [0., inf] <- naive form fails at both ends
8print(F.softplus(big)) # [2.0612e-09, 100.]
9
10def softplus(x, beta=1.0): # from scratch, stable
11 bx = beta * x
12 return (bx.clamp(min=0) + torch.log1p(torch.exp(-bx.abs()))) / beta
13
14sp = nn.Softplus(beta=1.0, threshold=20.0) # returns x itself where beta*x > 20
15
16# positive output head, initialised so that sigma starts at 1.0
17head = nn.Linear(64, 1)
18nn.init.zeros_(head.weight); nn.init.constant_(head.bias, torch.log(torch.expm1(torch.tensor(1.0))).item()) # 0.5413
19sigma = F.softplus(head(torch.randn(8, 64))) + 1e-6 # all ≈ 1.0
Module, functional, the overflow demo, and a from-scratch stable version

Use when

  • +Smooth, convex and strictly positive: the natural transform for scales, variances and step sizes.
  • +Derivative is the sigmoid, never exactly 0, so there are no dead units.
  • +β tunes it continuously toward ReLU, with an exact error bound ln 2/β.

Avoid when

  • −As a hidden activation it is beaten or matched by ReLU at a fraction of the cost.
  • −Not zero-centred (mean 0.806 for unit Gaussian inputs) and saturating on the negative side.
  • −The literal formula overflows above 88.7 and underflows below −16.6 in float32.

Check yourself

What is the largest gap between softplus_β and ReLU, and where does it occur?

Check yourself

In float32, the naive log(1 + exp(x)) returns 0 for x = −20. Why?

Remember

  • ✓softplus(x)=ln⁡(1+ex)\mathrm{softplus}(x)=\ln(1+e^x) (Dugas et al. 2001); softplus′=σ\mathrm{softplus}'=\sigma, softplus′′=σ(1−σ)>0\mathrm{softplus}''=\sigma(1-\sigma)>0.
  • ✓Gap to ReLU is ln⁡(1+e−∣x∣)\ln(1+e^{-|x|}), at most ln⁡2\ln2; with sharpness β\beta it is at most ln⁡2/β\ln2/\beta.
  • ✓Compute it as max⁡(x,0)+log1p⁡(e−∣x∣)\max(x,0)+\operatorname{log1p}(e^{-|x|}); the literal form fails above 88.72 and below −16.64 in float32.
  • ✓Its main modern role is positivity: Gaussian scales, Mamba's Δ, learned temperatures; initialise through ln(eʸ − 1).
  • ✓As a hidden activation ReLU matches or beats it (Glorot et al. 2011).
← back to Deep Learning roadmap