# Softplus

> log(1 + eˣ): a smooth, always-positive ReLU whose derivative is the sigmoid. It is rarely a hidden activation today, but it is the standard way to keep an output positive.

Source: https://www.mlfoundry.in/learn/softplus  
Section: Deep Learning › Activation Functions  
Level: core · 7 min read · updated 2026-10-11

**Softplus:** is the activation $\mathrm{softplus}(x)=\ln(1+e^{x})$. It is a smooth approximation of $\max(0,x)$ that is strictly positive everywhere, and its derivative is the logistic sigmoid $\sigma(x)$.

Softplus is the integral of the sigmoid: $\int_{-\infty}^{x}\sigma(t)\,dt=\ln(1+e^x)$. The sigmoid is a smooth step and ReLU's slope is a hard step, so softplus relates to ReLU the way the sigmoid relates to the step function.

Dugas, Bengio, Bélisle, Nadeau & Garcia introduced it as a neural-network nonlinearity in 2001, for option pricing, where a convex, positive, smooth function was the right prior. Glorot, Bordes & Bengio (2011) compared it with ReLU in deep networks and found the rectifier performed as well or better despite its hard zero and kink. ReLU won the hidden-layer role. Softplus kept another job: **turning an unconstrained number into a positive one**.

## Formula and derivative

$$
\mathrm{softplus}(x)=\ln(1+e^{x}),\qquad \mathrm{softplus}'(x)=\frac{e^x}{1+e^x}=\sigma(x),\qquad \mathrm{softplus}''(x)=\sigma(x)\big(1-\sigma(x)\big)
$$

- $\sigma(x)$: logistic sigmoid; the slope is in $(0,1)$ and never exactly 0
- $\sigma(1-\sigma)$: always positive: softplus is strictly convex, with peak curvature $0.25$ at $x=0$

Its distance from ReLU is $\mathrm{softplus}(x)-\max(0,x)=\ln(1+e^{-|x|})$. That gap is symmetric, largest at the origin ($\ln2=0.6931$) and decays like $e^{-|x|}$: $0.1269$ at $|x|=2$ and $0.0067$ at $|x|=5$.

## Sharpness β, and the stable form

A sharpness parameter $\beta$ trades smoothness for closeness to ReLU:

$$
\mathrm{softplus}_\beta(x)=\frac1\beta\ln\!\big(1+e^{\beta x}\big),\qquad \max_x\big|\mathrm{softplus}_\beta(x)-\max(0,x)\big|=\frac{\ln 2}{\beta}
$$

- $\beta$: sharpness; $\beta\to\infty$ gives ReLU, and the slope is $\sigma(\beta x)$
- $\ln2/\beta$: the largest gap to ReLU, at $x=0$; e.g. $0.0693$ for $\beta=10$

Written literally, $\ln(1+e^x)$ breaks in floating point at both ends. For large $x$, $e^x$ overflows to `inf`. For very negative $x$, $e^x$ is smaller than half the spacing of floats near 1, so $1+e^x$ rounds to exactly 1 and the log returns 0. The fix is an exact rewrite:

$$
\ln(1+e^{x})=\max(x,0)+\operatorname{log1p}\!\big(e^{-|x|}\big)
$$

- $e^{-|x|}$: always in $(0,1]$, so it cannot overflow
- $\operatorname{log1p}(u)$: $\ln(1+u)$ computed accurately for tiny $u$, where $\ln(1+u)\approx u$

In float32 the naive form returns `inf` for $x>88.72$ ($=\ln$ of the largest float32) and returns exactly 0 for $x<-16.64$ ($e^x<2^{-24}$). Between about $-16$ and $-5$ it returns a number with a large relative error. In float64 the limits move to $709.78$ and $-36.74$ but do not go away. The stable form is accurate to float32 rounding across the whole range.

> Figure: Top: drag β. The shaded gap to ReLU shrinks, and its maximum (at x = 0) is always log 2/β; on the right the slope σ(βx) sharpens into ReLU's step. Bottom: the relative error of each implementation, computed live with every operation rounded to float32. The naive form (rose) is 100% wrong left of −16.6 and infinite right of 88.7; the stable form (green) stays near 1e-7. Move the probe or jump to −20 and 89 to see the intermediate values.

> Figure: Softplus against ReLU, with the sigmoid for reference: the derivative panel shows softplus′ lying exactly on the sigmoid curve.

**At a glance**

- **Range:** $(0,\infty)$: strictly positive, unbounded above
- **Monotonic:** Yes, strictly increasing (slope σ(x) > 0)
- **Smooth:** Yes, infinitely differentiable and strictly convex
- **Zero-centred:** No. For $x\sim\mathcal N(0,1)$ the output has mean $0.806$ and std $0.521$
- **Saturation:** The slope saturates to 0 as $x\to-\infty$ (like the sigmoid) but never reaches it; no saturation above
- **Cost:** An exp and a log1p per element: far more than ReLU's compare

## Worked example

Softplus, its slope and its gap to ReLU (highlighted: the largest gap; the last column is the naive formula in float32)

| x | softplus(x) | σ(x) = slope | gap to ReLU | naive float32 |
| --- | --- | --- | --- | --- |
| −20 | 2.0612e−9 | 2.0612e−9 | 2.0612e−9 | 0 (wrong) |
| −5 | 0.0067 | 0.0067 | 0.0067 | 0.0067 |
| −1 | 0.3133 | 0.2689 | 0.3133 | 0.3133 |
| 0 | 0.6931 | 0.5000 | 0.6931 | 0.6931 |
| 1 | 1.3133 | 0.7311 | 0.3133 | 1.3133 |
| 5 | 5.0067 | 0.9933 | 0.0067 | 5.0067 |
| 20 | 20.0000 | 1.0000 | 2.0612e−9 | 20.0000 |
| 100 | 100.0000 | 1.0000 | ≈ 3.7e−44 | inf (wrong) |

For very negative $x$, $\mathrm{softplus}(x)\approx e^x\approx\sigma(x)$, which is why the second and third columns agree at $-20$. For large $x$, the output is $x$ and the slope is 1, exactly ReLU.

## Where it is used today

- **Positive scales and variances:** Gaussian output heads, mixture density networks and probabilistic regressors predict $\sigma=\mathrm{softplus}(a)+\varepsilon$. It grows linearly rather than exponentially, so a large raw value does not explode the scale the way $e^a$ can. VAEs often predict $\log\sigma^2$ instead; softplus is the common alternative.
- **Mamba's step size Δ:** Selective state-space models (Mamba, Gu & Dao 2023) compute their input-dependent discretisation step as $\Delta=\mathrm{softplus}(\text{linear}(x)+b)$, which keeps it positive. The bias is initialised through the inverse softplus so that $\Delta$ starts in a chosen range.
- **Constrained parameters:** Any learned quantity that must be positive (a temperature, a length-scale, a precision) can be stored as an unconstrained $a$ and used as $\mathrm{softplus}(a)$. To start at value $y$, initialise $a=\ln(e^{y}-1)$, e.g. $0.5413$ for $y=1$.
- **Inside losses:** Binary cross-entropy with logits is $\mathrm{softplus}(z)-yz$, and `F.binary_cross_entropy_with_logits` computes it with the same stable trick.

## Effect on training

As a hidden activation, softplus has no dead units (the slope is never 0), but it saturates on the negative side like a sigmoid. Its outputs are all positive with mean $0.806$ for unit-variance inputs, which biases the next layer's pre-activations. It is also convex, so a softplus unit cannot produce the small negative dip of GELU or Swish. These are the reasons ReLU-family and GELU-family activations displaced it. As an **output** transform its gradient $\sigma(a)$ is well behaved: close to 1 when the predicted value is large and decaying smoothly toward 0 as the value approaches its floor.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-2., -1., 0., 1., 2.])
print(F.softplus(x))                        # [0.1269, 0.3133, 0.6931, 1.3133, 2.1269]

big = torch.tensor([-20., 100.])            # float32
print(torch.log(1 + torch.exp(big)))        # [0., inf]          <- naive form fails at both ends
print(F.softplus(big))                      # [2.0612e-09, 100.]

def softplus(x, beta=1.0):                  # from scratch, stable
    bx = beta * x
    return (bx.clamp(min=0) + torch.log1p(torch.exp(-bx.abs()))) / beta

sp = nn.Softplus(beta=1.0, threshold=20.0)  # returns x itself where beta*x > 20

# positive output head, initialised so that sigma starts at 1.0
head = nn.Linear(64, 1)
nn.init.zeros_(head.weight); nn.init.constant_(head.bias, torch.log(torch.expm1(torch.tensor(1.0))).item())  # 0.5413
sigma = F.softplus(head(torch.randn(8, 64))) + 1e-6       # all ≈ 1.0
```

Module, functional, the overflow demo, and a from-scratch stable version

> **The threshold argument:** `nn.Softplus(beta, threshold=20)` switches to the identity where $\beta x>20$. That is a speed and safety shortcut, not an approximation you will notice: the dropped term is $\ln(1+e^{-20})/\beta=2.06\times10^{-9}/\beta$. If you write softplus yourself, never use the literal `log(1 + exp(x))`.

> **Softplus vs exp for positivity:** Both map $\mathbb{R}\to(0,\infty)$. $e^a$ gives multiplicative steps in $a$, which suits log-scale quantities but can overflow or swing wildly. $\mathrm{softplus}(a)$ is linear for large $a$ and exponential only near 0, so it is the safer default when the value can be large. Add a small $\varepsilon$ when the value is used as a divisor or inside a log.

**Strengths**

- Smooth, convex and strictly positive: the natural transform for scales, variances and step sizes.
- Derivative is the sigmoid, never exactly 0, so there are no dead units.
- β tunes it continuously toward ReLU, with an exact error bound ln 2/β.

**Limitations**

- As a hidden activation it is beaten or matched by ReLU at a fraction of the cost.
- Not zero-centred (mean 0.806 for unit Gaussian inputs) and saturating on the negative side.
- The literal formula overflows above 88.7 and underflows below −16.6 in float32.

**Question:** What is the largest gap between softplus_β and ReLU, and where does it occur?

- A. ln 2/β, at x = 0
- B. 1/β, at x = 0
- C. ln 2, at every x
- D. It grows without bound as x → ∞

**Answer:** A. ln 2/β, at x = 0 — The gap is $\tfrac1\beta\ln(1+e^{-\beta|x|})$. It is largest at $x=0$, $\ln2/\beta$, and decays exponentially on both sides. For $\beta=10$ it is $0.0693$.

**Question:** In float32, the naive log(1 + exp(x)) returns 0 for x = −20. Why?

- A. exp(−20) underflows to 0
- B. 1 + exp(−20) rounds to exactly 1, and log(1) = 0
- C. log of a number below 1 is negative and gets clamped
- D. Softplus is exactly 0 for x < −16

**Answer:** B. 1 + exp(−20) rounds to exactly 1, and log(1) = 0 — $e^{-20}=2.06\times10^{-9}$ is perfectly representable in float32, but it is below half the spacing of floats near 1 ($2^{-24}\approx5.96\times10^{-8}$), so the sum rounds to 1. `log1p(exp(x))` avoids forming $1+u$ and returns $2.06\times10^{-9}$.

**Key points**

- $\mathrm{softplus}(x)=\ln(1+e^x)$ (Dugas et al. 2001); $\mathrm{softplus}'=\sigma$, $\mathrm{softplus}''=\sigma(1-\sigma)>0$.
- Gap to ReLU is $\ln(1+e^{-|x|})$, at most $\ln2$; with sharpness $\beta$ it is at most $\ln2/\beta$.
- Compute it as $\max(x,0)+\operatorname{log1p}(e^{-|x|})$; the literal form fails above 88.72 and below −16.64 in float32.
- Its main modern role is positivity: Gaussian scales, Mamba's Δ, learned temperatures; initialise through ln(eʸ − 1).
- As a hidden activation ReLU matches or beats it (Glorot et al. 2011).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
