# Sigmoid

> σ(x) = 1/(1 + e^−x) turns a log-odds score into a probability, so it is the right output for binary decisions and gates. Its slope is at most 0.25, which makes it a poor hidden activation.

Source: https://www.mlfoundry.in/learn/sigmoid  
Section: Deep Learning › Activation Functions  
Level: core · 10 min read · updated 2026-10-11

**Sigmoid:** (the **logistic function**) is $\sigma(x)=1/(1+e^{-x})$. It maps any real number to $(0,1)$, with $\sigma(0)=\tfrac12$, and is the inverse of the log-odds (logit) function, so it reads a score as a probability.

The logistic curve comes from 19th-century population models (Verhulst). In neural networks it was the smooth, differentiable replacement for the threshold unit: Rumelhart, Hinton and Williams used it in their 1986 backpropagation paper, and for two decades it was the default hidden activation. Cybenko's 1989 universal approximation theorem was stated for sigmoidal functions.

Today it has left the hidden layers, replaced first by tanh and then by ReLU, but it is everywhere else: binary and multi-label outputs, LSTM and GRU gates, attention-style gating (squeeze-and-excitation), and inside SiLU/Swish, $x\,\sigma(x)$.

## Formula, derivative and inverse

$$
\sigma(x)=\frac{1}{1+e^{-x}}=\frac{e^{x}}{1+e^{x}},\qquad \sigma'(x)=\sigma(x)\big(1-\sigma(x)\big)\le\tfrac14
$$

- $x$: the pre-activation (logit, score)
- $\sigma'$: computed from the output alone; maximum 1/4 at x = 0
- $1-\sigma(x)$: equals $\sigma(-x)$: the curve is point-symmetric about $(0,\tfrac12)$

$$
\sigma^{-1}(p)=\operatorname{logit}(p)=\log\frac{p}{1-p},\qquad \sigma(x)=\tfrac12\big(1+\tanh\tfrac x2\big),\qquad \log\sigma(x)=-\operatorname{softplus}(-x)
$$

- $\operatorname{logit}$: log-odds: p = 0.9 has logit 2.197, p = 0.99 has logit 4.595
- $\tanh$: a sigmoid is a shifted, rescaled tanh
- $\operatorname{softplus}(u)$: $\log(1+e^{u})$, the stable way to compute log-probabilities

**Properties**

- **Range:** $(0,1)$, never reaching 0 or 1
- **Monotonic:** strictly increasing
- **Smooth:** yes ($C^\infty$)
- **Zero-centred:** no: $\sigma(0)=\tfrac12$ and every output is positive
- **Saturating:** both sides: $\sigma'(4)=0.0177$, $\sigma'(10)\approx4.5\times10^{-5}$
- **Max slope:** $\tfrac14$ at $x=0$
- **Cost:** one exp and one division

> Figure: The sigmoid and its bell-shaped derivative, capped at 0.25, with tanh for comparison: same S-shape, but range (−1, 1) and slope 1 at the origin.

## A probability for binary outcomes

Suppose a model outputs a score $z$ and we want $p=P(y=1\mid x)$. If $z$ is the **log-odds**, $z=\log\frac{p}{1-p}$, then solving for $p$ gives exactly $p=\sigma(z)$. That is logistic regression, and a network with a sigmoid output is logistic regression on learned features.

Paired with binary cross-entropy, the output gradient is remarkably simple:

$$
\mathcal{L}=-\big[y\log p+(1-y)\log(1-p)\big],\quad p=\sigma(z)\quad\Longrightarrow\quad \frac{\partial\mathcal{L}}{\partial z}=p-y
$$

- $y\in\{0,1\}$: the label
- $p-y$: the $\sigma'$ in the chain rule cancels against the $1/p$ and $1/(1-p)$ of the log

With squared error instead, $\partial\mathcal{L}/\partial z=2(p-y)\,\sigma'(z)$ keeps the $\sigma'$ factor. A confidently wrong prediction ($y=1$, $z=-4$) then gets almost no gradient: $0.0347$, versus $0.982$ for cross-entropy. That is why sigmoid outputs are trained with cross-entropy.

## Why it fails as a hidden activation

**Small slope.** Backprop multiplies by $\sigma'(z)\le\tfrac14$ at every layer. With weights of norm about 1, $L$ layers scale the gradient by at most $0.25^{L}$: $9.8\times10^{-4}$ after 5 layers, $9.5\times10^{-7}$ after 10. That is the classic **vanishing gradient**.

**Saturation.** For $|z|$ beyond about 4 the curve is flat, so $\sigma'(z)$ is close to zero. Larger weights push more pre-activations there: they help the product of weights, but they switch off the slope.

**Not zero-centred.** Every output is positive, with mean about $0.5$. The next layer then sees all-positive inputs, which forces all its weight updates for one unit to share a sign and makes optimization zig-zag (see **Tanh**). Glorot and Bengio (2010) also observed that with sigmoid units and standard initialization, the top hidden layer quickly saturates at 0, and training is slow to recover from it.

> Figure: Top left: a real forward and backward pass through 10 sigmoid layers of 16 units (W ~ N(0, s²/16), 32 inputs). At s = 1 the gradient at layer 1 is 2.1e−6 of the gradient at layer 10, ×0.233 per layer, close to the 0.25 bound (rose). Raise s to 4: the decay slows to 9.9e−3 overall, but the histogram at layer 10 shows 19% of units saturated. At s = 8 it is 0.25, with 59% of layer-10 units saturated and their mean slope down to 0.056. Larger weights offset the 0.25 by switching units off. Top right: where the selected layer's pre-activations fall on σ′. Bottom: the output gradient for a positive example. Cross-entropy (accent) stays near 1 when the model is confidently wrong; squared error (rose) collapses to 0.035 at z = −4.

## Worked example

σ and σ′ at a few points (computed in Node)

| x | −4 | −2 | −1 | 0 | 1 | 2 | 4 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| σ(x) | 0.0180 | 0.1192 | 0.2689 | 0.5000 | 0.7311 | 0.8808 | 0.9820 |
| σ′(x) | 0.0177 | 0.1050 | 0.1966 | 0.2500 | 0.1966 | 0.1050 | 0.0177 |

A single output unit with $w=2$, $b=-1$, input $x=1.5$ and label $y=0$: $z=2\cdot1.5-1=2$, $p=\sigma(2)=0.8808$, cross-entropy $-\log(1-0.8808)=2.1269$. Backward: $\partial\mathcal{L}/\partial z=p-y=0.8808$, so $\partial\mathcal{L}/\partial w=0.8808\cdot1.5=1.3212$ and $\partial\mathcal{L}/\partial b=0.8808$.

Compare a chain of three sigmoid units with weight 1.5 each, starting from $x=0.5$. The pre-activations are $0.750$, $1.019$, $1.102$, and the end-to-end derivative is $0.0268$. The same chain with tanh gives $0.320$, about 12 times more.

## Numerical stability

$e^{-x}$ overflows for large negative $x$: in float32, $e^{u}$ is infinite once $u>88.72$. The sigmoid itself survives (it becomes $1/\infty=0$), but its logarithm does not. In float64, $\log\sigma(-800)$ evaluates to $-\infty$ instead of $-800$. And $1-\sigma(40)$ rounds to exactly 0, while $\sigma(-40)=4.2\times10^{-18}$.

**Rules for a stable sigmoid**

1. Compute $\sigma(x)$ as $1/(1+e^{-x})$ for $x\ge0$ and as $e^{x}/(1+e^{x})$ for $x<0$, so the exponent is never positive.
2. Never take $\log$ of a sigmoid output. Use $\log\sigma(x)=-\operatorname{softplus}(-x)$, where $\operatorname{softplus}(u)=\max(u,0)+\log(1+e^{-|u|})$.
3. Use $\sigma(-x)$ rather than $1-\sigma(x)$ for the complement.
4. In PyTorch, have the model output logits and use `BCEWithLogitsLoss`, which fuses the sigmoid and the log.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-800.0, -4.0, 0.0, 4.0, 800.0])

act = nn.Sigmoid()                 # module
p = torch.sigmoid(x)               # functional (stable internally)

def sigmoid(x):                    # from scratch: exponent is never positive
    e = torch.exp(-x.abs())
    return torch.where(x >= 0, 1 / (1 + e), e / (1 + e))

def dsigmoid(x):
    s = sigmoid(x)
    return s * (1 - s)             # max 0.25 at x = 0

# Binary classifier: output a logit, let the loss apply the sigmoid
logits = torch.tensor([-800.0, 3.0])
y = torch.tensor([1.0, 1.0])
print(F.binary_cross_entropy_with_logits(logits, y, reduction="none"))  # [800.0000, 0.0486]
print(-torch.log(torch.sigmoid(logits)) * y)                            # [inf, 0.0486]  ← naive

# Multi-label: one independent sigmoid per class (not a softmax)
multi = nn.Sequential(nn.Linear(512, 20))   # train with nn.BCEWithLogitsLoss()
```

Sigmoid in PyTorch: module, functional, a stable from-scratch version, and the fused loss

> **Sigmoid + BCELoss on probabilities:** `nn.Sequential(..., nn.Sigmoid())` followed by `nn.BCELoss()` works until a prediction saturates. Then $p$ rounds to exactly 0 or 1, $\log p$ becomes $\log 0$ (PyTorch clamps it at $-100$), and the loss and its gradient no longer reflect how wrong the logit is. Drop the final sigmoid, use `nn.BCEWithLogitsLoss`, and apply `torch.sigmoid` only when you need probabilities at inference.

> **Why gates use sigmoids:** A gate multiplies a signal by a number that should mean "fraction let through", so it must lie in $(0,1)$ and be differentiable. LSTMs (Hochreiter & Schmidhuber, 1997; forget gate added by Gers et al., 2000), GRUs and highway networks all use $\sigma$ for this. A forget gate near 1 keeps the cell state, and its gradient, almost unchanged across many steps.

**Strengths**

- Output is a probability: the natural link for binary and multi-label targets
- With cross-entropy the output gradient is just $p-y$
- Bounded and smooth: ideal for gates
- Derivative is cheap: $\sigma(1-\sigma)$ from the forward value

**Limitations**

- Max slope $\tfrac14$: gradients shrink at least $4\times$ per layer
- Saturates on both sides, so large pre-activations stop learning
- Outputs all positive (mean ≈ 0.5): zig-zag updates in the next layer
- Needs care with exp and log for numerical stability

**Question:** A sigmoid output unit with target y = 1 outputs z = −3 (p = 0.047). What is ∂L/∂z under binary cross-entropy, and under squared error (p − y)²?

- A. −0.953 and −0.086
- B. −0.086 and −0.953
- C. −0.25 and −0.25
- D. 0 and 0, because the unit is saturated

**Answer:** A. −0.953 and −0.086 — Cross-entropy: $p-y=0.0474-1=-0.9526$. Squared error: $2(p-y)\,\sigma'(z)=2(-0.9526)(0.0474\cdot0.9526)=-0.0861$. Squared error keeps the $\sigma'$ factor, so it learns least when the prediction is most wrong.

**Question:** Which identity lets you compute log σ(x) without overflow?

- A. log σ(x) = −log(1 + e^x)
- B. log σ(x) = −softplus(−x)
- C. log σ(x) = x − 1
- D. log σ(x) = 1 − log(1 + e^−x)

**Answer:** B. log σ(x) = −softplus(−x) — $\log\sigma(x)=-\log(1+e^{-x})=-\operatorname{softplus}(-x)$, and softplus has the stable form $\max(u,0)+\log(1+e^{-|u|})$. At $x=-800$ it returns $-800$; computing $\log(\sigma(-800))$ directly gives $-\infty$.

**Key points**

- $\sigma(x)=1/(1+e^{-x})\in(0,1)$, inverse of the logit; $\sigma'=\sigma(1-\sigma)\le\tfrac14$.
- Use it where a probability or a gate is needed: binary / multi-label outputs, LSTM and GRU gates, SE blocks, inside SiLU.
- Sigmoid + cross-entropy gives $\partial\mathcal{L}/\partial z=p-y$. Sigmoid + squared error stalls on confident mistakes.
- As a hidden activation it vanishes ($0.25^{10}\approx9.5\times10^{-7}$), saturates for $|z|\gtrsim4$, and is not zero-centred.
- Compute in logit space: `BCEWithLogitsLoss`, $\log\sigma(x)=-\operatorname{softplus}(-x)$, $\sigma(-x)$ instead of $1-\sigma(x)$.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
