σ(x) = 1/(1 + e^−x) turns a log-odds score into a probability, so it is the right output for binary decisions and gates. Its slope is at most 0.25, which makes it a poor hidden activation.
10 min read685 wordscore0 viewsupdated 2026-10-11
Definition
Sigmoid (the logistic function) is σ(x)=1/(1+e−x). It maps any real number to (0,1), with σ(0)=21, and is the inverse of the log-odds (logit) function, so it reads a score as a probability.
The logistic curve comes from 19th-century population models (Verhulst). In neural networks it was the smooth, differentiable replacement for the threshold unit: Rumelhart, Hinton and Williams used it in their 1986 backpropagation paper, and for two decades it was the default hidden activation. Cybenko's 1989 universal approximation theorem was stated for sigmoidal functions.
Today it has left the hidden layers, replaced first by tanh and then by ReLU, but it is everywhere else: binary and multi-label outputs, LSTM and GRU gates, attention-style gating (squeeze-and-excitation), and inside SiLU/Swish, xσ(x).
Formula, derivative and inverse
σ(x)=1+e−x1=1+exex,σ′(x)=σ(x)(1−σ(x))≤41
x
the pre-activation (logit, score)
σ′
computed from the output alone; maximum 1/4 at x = 0
1−σ(x)
equals σ(−x): the curve is point-symmetric about (0,21)
log-odds: p = 0.9 has logit 2.197, p = 0.99 has logit 4.595
tanh
a sigmoid is a shifted, rescaled tanh
softplus(u)
log(1+eu), the stable way to compute log-probabilities
Properties
Range
(0,1), never reaching 0 or 1
Monotonic
strictly increasing
Smooth
yes (C∞)
Zero-centred
no: σ(0)=21 and every output is positive
Saturating
both sides: σ′(4)=0.0177, σ′(10)≈4.5×10−5
Max slope
41 at x=0
Cost
one exp and one division
activation
compare
range (0, 1)monotonic yessmooth yeszero-centred no
Maximum slope 0.25 at x = 0; saturates for large |x|.
E[f(x)] 0.500Var[f(x)] 0.043E[f′(x)] 0.207f′ = 0 for 0.0% of inputsE[f′]¹⁰ 1.4e−7
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.The sigmoid and its bell-shaped derivative, capped at 0.25, with tanh for comparison: same S-shape, but range (−1, 1) and slope 1 at the origin.
A probability for binary outcomes
Suppose a model outputs a score z and we want p=P(y=1∣x). If z is the log-odds, z=log1−pp, then solving for p gives exactly p=σ(z). That is logistic regression, and a network with a sigmoid output is logistic regression on learned features.
Paired with binary cross-entropy, the output gradient is remarkably simple:
L=−[ylogp+(1−y)log(1−p)],p=σ(z)⟹∂z∂L=p−y
y∈{0,1}
the label
p−y
the σ′ in the chain rule cancels against the 1/p and 1/(1−p) of the log
With squared error instead, ∂L/∂z=2(p−y)σ′(z) keeps the σ′ factor. A confidently wrong prediction (y=1, z=−4) then gets almost no gradient: 0.0347, versus 0.982 for cross-entropy. That is why sigmoid outputs are trained with cross-entropy.
Why it fails as a hidden activation
Small slope. Backprop multiplies by σ′(z)≤41 at every layer. With weights of norm about 1, L layers scale the gradient by at most 0.25L: 9.8×10−4 after 5 layers, 9.5×10−7 after 10. That is the classic vanishing gradient.
Saturation. For ∣z∣ beyond about 4 the curve is flat, so σ′(z) is close to zero. Larger weights push more pre-activations there: they help the product of weights, but they switch off the slope.
Not zero-centred. Every output is positive, with mean about 0.5. The next layer then sees all-positive inputs, which forces all its weight updates for one unit to share a sign and makes optimization zig-zag (see Tanh). Glorot and Bengio (2010) also observed that with sigmoid units and standard initialization, the top hidden layer quickly saturates at 0, and training is slow to recover from it.
W ~ N(0, s²/16)
fig.Top left: a real forward and backward pass through 10 sigmoid layers of 16 units (W ~ N(0, s²/16), 32 inputs). At s = 1 the gradient at layer 1 is 2.1e−6 of the gradient at layer 10, ×0.233 per layer, close to the 0.25 bound (rose). Raise s to 4: the decay slows to 9.9e−3 overall, but the histogram at layer 10 shows 19% of units saturated. At s = 8 it is 0.25, with 59% of layer-10 units saturated and their mean slope down to 0.056. Larger weights offset the 0.25 by switching units off. Top right: where the selected layer's pre-activations fall on σ′. Bottom: the output gradient for a positive example. Cross-entropy (accent) stays near 1 when the model is confidently wrong; squared error (rose) collapses to 0.035 at z = −4.
Worked example
x
−4
−2
−1
0
1
2
4
σ(x)
0.0180
0.1192
0.2689
0.5000
0.7311
0.8808
0.9820
σ′(x)
0.0177
0.1050
0.1966
0.2500
0.1966
0.1050
0.0177
σ and σ′ at a few points (computed in Node)
A single output unit with w=2, b=−1, input x=1.5 and label y=0: z=2⋅1.5−1=2, p=σ(2)=0.8808, cross-entropy −log(1−0.8808)=2.1269. Backward: ∂L/∂z=p−y=0.8808, so ∂L/∂w=0.8808⋅1.5=1.3212 and ∂L/∂b=0.8808.
Compare a chain of three sigmoid units with weight 1.5 each, starting from x=0.5. The pre-activations are 0.750, 1.019, 1.102, and the end-to-end derivative is 0.0268. The same chain with tanh gives 0.320, about 12 times more.
Numerical stability
e−x overflows for large negative x: in float32, eu is infinite once u>88.72. The sigmoid itself survives (it becomes 1/∞=0), but its logarithm does not. In float64, logσ(−800) evaluates to −∞ instead of −800. And 1−σ(40) rounds to exactly 0, while σ(−40)=4.2×10−18.
Rules for a stable sigmoid
1Compute σ(x) as 1/(1+e−x) for x≥0 and as ex/(1+ex) for x<0, so the exponent is never positive.
2Never take log of a sigmoid output. Use logσ(x)=−softplus(−x), where softplus(u)=max(u,0)+log(1+e−∣u∣).
3Use σ(−x) rather than 1−σ(x) for the complement.
4In PyTorch, have the model output logits and use BCEWithLogitsLoss, which fuses the sigmoid and the log.
python
1import torch, torch.nn as nn, torch.nn.functional as F