Sigmoid

σ(x) = 1/(1 + e^−x) turns a log-odds score into a probability, so it is the right output for binary decisions and gates. Its slope is at most 0.25, which makes it a poor hidden activation.

10 min read685 wordscore0 viewsupdated 2026-10-11

Definition

Sigmoid (the logistic function) is σ(x)=1/(1+e−x)\sigma(x)=1/(1+e^{-x}). It maps any real number to (0,1)(0,1), with σ(0)=12\sigma(0)=\tfrac12, and is the inverse of the log-odds (logit) function, so it reads a score as a probability.

The logistic curve comes from 19th-century population models (Verhulst). In neural networks it was the smooth, differentiable replacement for the threshold unit: Rumelhart, Hinton and Williams used it in their 1986 backpropagation paper, and for two decades it was the default hidden activation. Cybenko's 1989 universal approximation theorem was stated for sigmoidal functions.

Today it has left the hidden layers, replaced first by tanh and then by ReLU, but it is everywhere else: binary and multi-label outputs, LSTM and GRU gates, attention-style gating (squeeze-and-excitation), and inside SiLU/Swish, x σ(x)x\,\sigma(x).

Formula, derivative and inverse

σ(x)=11+e−x=ex1+ex,σ′(x)=σ(x)(1−σ(x))≤14\sigma(x)=\frac{1}{1+e^{-x}}=\frac{e^{x}}{1+e^{x}},\qquad \sigma'(x)=\sigma(x)\big(1-\sigma(x)\big)\le\tfrac14
xx
the pre-activation (logit, score)
σ′\sigma'
computed from the output alone; maximum 1/4 at x = 0
1−σ(x)1-\sigma(x)
equals σ(−x)\sigma(-x): the curve is point-symmetric about (0,12)(0,\tfrac12)
σ−1(p)=logit⁡(p)=log⁡p1−p,σ(x)=12(1+tanh⁡x2),log⁡σ(x)=−softplus⁡(−x)\sigma^{-1}(p)=\operatorname{logit}(p)=\log\frac{p}{1-p},\qquad \sigma(x)=\tfrac12\big(1+\tanh\tfrac x2\big),\qquad \log\sigma(x)=-\operatorname{softplus}(-x)
logit⁡\operatorname{logit}
log-odds: p = 0.9 has logit 2.197, p = 0.99 has logit 4.595
tanh⁡\tanh
a sigmoid is a shifted, rescaled tanh
softplus⁡(u)\operatorname{softplus}(u)
log⁡(1+eu)\log(1+e^{u}), the stable way to compute log-probabilities

Properties

Range
(0,1)(0,1), never reaching 0 or 1
Monotonic
strictly increasing
Smooth
yes (C∞C^\infty)
Zero-centred
no: σ(0)=12\sigma(0)=\tfrac12 and every output is positive
Saturating
both sides: σ′(4)=0.0177\sigma'(4)=0.0177, σ′(10)≈4.5×10−5\sigma'(10)\approx4.5\times10^{-5}
Max slope
14\tfrac14 at x=0x=0
Cost
one exp and one division
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 0.731 · f′(x) = 0.197−1.0−0.50.00.51.0−4−2024x →SigmoidSigmoid f′Tanh
activation
compare
range (0, 1)monotonic yessmooth yeszero-centred no
Maximum slope 0.25 at x = 0; saturates for large |x|.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]0.020.500.98output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.500Var[f(x)] 0.043E[f′(x)] 0.207f′ = 0 for 0.0% of inputsE[f′]¹⁰ 1.4e−7
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.The sigmoid and its bell-shaped derivative, capped at 0.25, with tanh for comparison: same S-shape, but range (−1, 1) and slope 1 at the origin.

A probability for binary outcomes

Suppose a model outputs a score zz and we want p=P(y=1∣x)p=P(y=1\mid x). If zz is the log-odds, z=log⁡p1−pz=\log\frac{p}{1-p}, then solving for pp gives exactly p=σ(z)p=\sigma(z). That is logistic regression, and a network with a sigmoid output is logistic regression on learned features.

Paired with binary cross-entropy, the output gradient is remarkably simple:

L=−[ylog⁡p+(1−y)log⁡(1−p)],p=σ(z)⟹∂L∂z=p−y\mathcal{L}=-\big[y\log p+(1-y)\log(1-p)\big],\quad p=\sigma(z)\quad\Longrightarrow\quad \frac{\partial\mathcal{L}}{\partial z}=p-y
y∈{0,1}y\in\{0,1\}
the label
p−yp-y
the σ′\sigma' in the chain rule cancels against the 1/p1/p and 1/(1−p)1/(1-p) of the log

With squared error instead, ∂L/∂z=2(p−y) σ′(z)\partial\mathcal{L}/\partial z=2(p-y)\,\sigma'(z) keeps the σ′\sigma' factor. A confidently wrong prediction (y=1y=1, z=−4z=-4) then gets almost no gradient: 0.03470.0347, versus 0.9820.982 for cross-entropy. That is why sigmoid outputs are trained with cross-entropy.

Why it fails as a hidden activation

Small slope. Backprop multiplies by σ′(z)≤14\sigma'(z)\le\tfrac14 at every layer. With weights of norm about 1, LL layers scale the gradient by at most 0.25L0.25^{L}: 9.8×10−49.8\times10^{-4} after 5 layers, 9.5×10−79.5\times10^{-7} after 10. That is the classic vanishing gradient.

Saturation. For ∣z∣|z| beyond about 4 the curve is flat, so σ′(z)\sigma'(z) is close to zero. Larger weights push more pre-activations there: they help the product of weights, but they switch off the slope.

Not zero-centred. Every output is positive, with mean about 0.50.5. The next layer then sees all-positive inputs, which forces all its weight updates for one unit to share a sign and makes optimization zig-zag (see Tanh). Glorot and Bengio (2010) also observed that with sigmoid units and standard initialization, the top hidden layer quickly saturates at 0, and training is slow to recover from it.

‖δ⁽ˡ⁾‖ / ‖δ⁽ᴸ⁾‖ per layer (log)10 sigmoid layers × 16 units, batch 3211e-41e-81e-12layer 1layer 10 (output side)╌ 0.25 per layer (σ′ at its max, ‖W‖ = 1)pre-activations z at layer 100% saturated (σ′ < 0.025) · mean σ′ 0.230−10−50510━ σ′(z)/0.25 · shaded: |z| > 3.64output unit with target y = 1: |∂L/∂z| against its logit z00.51−8−4048← confidently wrongconfidently right →cross-entropy: p − y = −(1 − p)squared error: 2(p − y)·σ′(z)at z = -4.0: p = 0.0180 · CE gradient 0.982 · MSE gradient 0.0347hidden stack: ‖δ⁽¹⁾‖/‖δ⁽ᴸ⁾‖ = 2.1e-6 · per layer ×0.233 · bound 0.25^9 = 3.8e-6
W ~ N(0, s²/16)
fig.Top left: a real forward and backward pass through 10 sigmoid layers of 16 units (W ~ N(0, s²/16), 32 inputs). At s = 1 the gradient at layer 1 is 2.1e−6 of the gradient at layer 10, ×0.233 per layer, close to the 0.25 bound (rose). Raise s to 4: the decay slows to 9.9e−3 overall, but the histogram at layer 10 shows 19% of units saturated. At s = 8 it is 0.25, with 59% of layer-10 units saturated and their mean slope down to 0.056. Larger weights offset the 0.25 by switching units off. Top right: where the selected layer's pre-activations fall on σ′. Bottom: the output gradient for a positive example. Cross-entropy (accent) stays near 1 when the model is confidently wrong; squared error (rose) collapses to 0.035 at z = −4.

Worked example

x−4−2−10124
σ(x)0.01800.11920.26890.50000.73110.88080.9820
σ′(x)0.01770.10500.19660.25000.19660.10500.0177
σ and σ′ at a few points (computed in Node)

A single output unit with w=2w=2, b=−1b=-1, input x=1.5x=1.5 and label y=0y=0: z=2⋅1.5−1=2z=2\cdot1.5-1=2, p=σ(2)=0.8808p=\sigma(2)=0.8808, cross-entropy −log⁡(1−0.8808)=2.1269-\log(1-0.8808)=2.1269. Backward: ∂L/∂z=p−y=0.8808\partial\mathcal{L}/\partial z=p-y=0.8808, so ∂L/∂w=0.8808⋅1.5=1.3212\partial\mathcal{L}/\partial w=0.8808\cdot1.5=1.3212 and ∂L/∂b=0.8808\partial\mathcal{L}/\partial b=0.8808.

Compare a chain of three sigmoid units with weight 1.5 each, starting from x=0.5x=0.5. The pre-activations are 0.7500.750, 1.0191.019, 1.1021.102, and the end-to-end derivative is 0.02680.0268. The same chain with tanh gives 0.3200.320, about 12 times more.

Numerical stability

e−xe^{-x} overflows for large negative xx: in float32, eue^{u} is infinite once u>88.72u>88.72. The sigmoid itself survives (it becomes 1/∞=01/\infty=0), but its logarithm does not. In float64, log⁡σ(−800)\log\sigma(-800) evaluates to −∞-\infty instead of −800-800. And 1−σ(40)1-\sigma(40) rounds to exactly 0, while σ(−40)=4.2×10−18\sigma(-40)=4.2\times10^{-18}.

Rules for a stable sigmoid

  1. 1Compute σ(x)\sigma(x) as 1/(1+e−x)1/(1+e^{-x}) for x≥0x\ge0 and as ex/(1+ex)e^{x}/(1+e^{x}) for x<0x<0, so the exponent is never positive.
  2. 2Never take log⁡\log of a sigmoid output. Use log⁡σ(x)=−softplus⁡(−x)\log\sigma(x)=-\operatorname{softplus}(-x), where softplus⁡(u)=max⁡(u,0)+log⁡(1+e−∣u∣)\operatorname{softplus}(u)=\max(u,0)+\log(1+e^{-|u|}).
  3. 3Use σ(−x)\sigma(-x) rather than 1−σ(x)1-\sigma(x) for the complement.
  4. 4In PyTorch, have the model output logits and use BCEWithLogitsLoss, which fuses the sigmoid and the log.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-800.0, -4.0, 0.0, 4.0, 800.0])
4
5act = nn.Sigmoid() # module
6p = torch.sigmoid(x) # functional (stable internally)
7
8def sigmoid(x): # from scratch: exponent is never positive
9 e = torch.exp(-x.abs())
10 return torch.where(x >= 0, 1 / (1 + e), e / (1 + e))
11
12def dsigmoid(x):
13 s = sigmoid(x)
14 return s * (1 - s) # max 0.25 at x = 0
15
16# Binary classifier: output a logit, let the loss apply the sigmoid
17logits = torch.tensor([-800.0, 3.0])
18y = torch.tensor([1.0, 1.0])
19print(F.binary_cross_entropy_with_logits(logits, y, reduction="none")) # [800.0000, 0.0486]
20print(-torch.log(torch.sigmoid(logits)) * y) # [inf, 0.0486] ← naive
21
22# Multi-label: one independent sigmoid per class (not a softmax)
23multi = nn.Sequential(nn.Linear(512, 20)) # train with nn.BCEWithLogitsLoss()
Sigmoid in PyTorch: module, functional, a stable from-scratch version, and the fused loss

Use when

  • +Output is a probability: the natural link for binary and multi-label targets
  • +With cross-entropy the output gradient is just p−yp-y
  • +Bounded and smooth: ideal for gates
  • +Derivative is cheap: σ(1−σ)\sigma(1-\sigma) from the forward value

Avoid when

  • −Max slope 14\tfrac14: gradients shrink at least 4×4\times per layer
  • −Saturates on both sides, so large pre-activations stop learning
  • −Outputs all positive (mean ≈ 0.5): zig-zag updates in the next layer
  • −Needs care with exp and log for numerical stability

Check yourself

A sigmoid output unit with target y = 1 outputs z = −3 (p = 0.047). What is ∂L/∂z under binary cross-entropy, and under squared error (p − y)²?

Check yourself

Which identity lets you compute log σ(x) without overflow?

Remember

  • ✓σ(x)=1/(1+e−x)∈(0,1)\sigma(x)=1/(1+e^{-x})\in(0,1), inverse of the logit; σ′=σ(1−σ)≤14\sigma'=\sigma(1-\sigma)\le\tfrac14.
  • ✓Use it where a probability or a gate is needed: binary / multi-label outputs, LSTM and GRU gates, SE blocks, inside SiLU.
  • ✓Sigmoid + cross-entropy gives ∂L/∂z=p−y\partial\mathcal{L}/\partial z=p-y. Sigmoid + squared error stalls on confident mistakes.
  • ✓As a hidden activation it vanishes (0.2510≈9.5×10−70.25^{10}\approx9.5\times10^{-7}), saturates for ∣z∣≳4|z|\gtrsim4, and is not zero-centred.
  • ✓Compute in logit space: BCEWithLogitsLoss, log⁡σ(x)=−softplus⁡(−x)\log\sigma(x)=-\operatorname{softplus}(-x), σ(−x)\sigma(-x) instead of 1−σ(x)1-\sigma(x).
← back to Deep Learning roadmap