# Softmax

> Exponentiate and normalise: turns a vector of logits into a probability distribution whose cross-entropy gradient is simply p − y.

Source: https://www.mlfoundry.in/learn/softmax  
Section: Deep Learning › Activation Functions  
Level: core · 9 min read · updated 2026-10-11

**Softmax:** maps a vector of real scores (logits) $z\in\mathbb{R}^K$ to a probability vector $p\in\Delta^{K-1}$ by exponentiating each score and dividing by the sum: $p_i=e^{z_i}/\sum_j e^{z_j}$. Every $p_i$ is positive and they sum to 1. It is a smooth, differentiable stand-in for $\arg\max$.

Most activations act on one number at a time. Softmax does not: each output depends on **every** input, because they all share the denominator. Raise one logit and its probability goes up while all the others go down. This coupling is what makes it a distribution, and it is why its derivative is a matrix, not a scalar.

It sits at the end of almost every classifier (one logit per class) and inside every attention layer (one score per key). The output layer of a language model is a softmax over the whole vocabulary.

## Formula

$$
\operatorname{softmax}(z)_i=\frac{e^{z_i/T}}{\sum_{j=1}^{K}e^{z_j/T}},\qquad \log\operatorname{softmax}(z)_i=\frac{z_i}{T}-\operatorname{LSE}\!\left(\frac{z}{T}\right)
$$

- $z_i$: logit (unnormalised score) of class i
- $K$: number of classes, keys or vocabulary tokens
- $T$: temperature, $T>0$; $T=1$ is the plain softmax
- $\operatorname{LSE}(z)$: log-sum-exp $\log\sum_j e^{z_j}$, a smooth maximum: $\max_j z_j\le\operatorname{LSE}(z)\le\max_j z_j+\log K$

**Shift invariance.** Adding the same constant $c$ to every logit multiplies numerator and denominator by $e^{c}$, so nothing changes: $\operatorname{softmax}(z+c\mathbf 1)=\operatorname{softmax}(z)$. Only differences between logits matter. Every implementation uses this to subtract $m=\max_j z_j$ first, so the largest exponent is $e^0=1$ and nothing can overflow:

$$
p_i=\frac{e^{z_i-m}}{\sum_j e^{z_j-m}},\qquad \operatorname{LSE}(z)=m+\log\sum_j e^{z_j-m}
$$

- $m$: $\max_j z_j$; any constant works mathematically, the max is the one that keeps every exponent $\le0$

**Two classes is a sigmoid.** With logits $(z_1,z_2)$, $p_1=1/(1+e^{-(z_1-z_2)})=\sigma(z_1-z_2)$. Fix $z_2=0$ and the softmax is exactly the logistic sigmoid, e.g. $\operatorname{softmax}(1.5,0)_1=\sigma(1.5)=0.8176$. Softmax is the multi-class generalisation of logistic regression's output.

## Derivative: the Jacobian

Differentiate $p_i$ with respect to each logit $z_j$. The diagonal term comes from the numerator, the off-diagonal ones from the shared denominator:

$$
\frac{\partial p_i}{\partial z_j}=\frac{1}{T}\,p_i\big(\delta_{ij}-p_j\big),\qquad J=\frac{1}{T}\Big(\operatorname{diag}(p)-p\,p^{\top}\Big)
$$

- $\delta_{ij}$: Kronecker delta: 1 if $i=j$, else 0
- $p_i(1-p_i)$: diagonal entries, positive, at most $0.25$ (at $p_i=0.5$), the sigmoid's slope again
- $-p_ip_j$: off-diagonal entries, always negative: raising one logit lowers every other probability

$J$ is symmetric and positive semidefinite, and every row and column sums to zero, because $\sum_i p_i=1$ cannot change. Its null space is the direction $\mathbf 1$: the matrix form of shift invariance.

In practice you never build $J$. Softmax is followed by cross-entropy $\mathcal L=-\log p_y$, and the product $J^\top\nabla_p\mathcal L$ collapses to something very simple:

$$
\mathcal L=-\log p_y=\operatorname{LSE}(z)-z_y\quad\Longrightarrow\quad \frac{\partial\mathcal L}{\partial z}=p-e_y\quad(T=1)
$$

- $e_y$: one-hot vector of the true class $y$
- $p-e_y$: predicted minus target: $p_y-1<0$ for the true class (push its logit up), $+p_i$ for every other class (push down)

**Properties**

- **Input → output:** $\mathbb{R}^K\to$ the open probability simplex: every $p_i\in(0,1)$, $\sum_i p_i=1$
- **Not elementwise:** each $p_i$ depends on all $z_j$; the derivative is a $K\times K$ Jacobian
- **Invariance:** $\operatorname{softmax}(z+c\mathbf 1)=\operatorname{softmax}(z)$; scaling is **not** invariant: $\operatorname{softmax}(az)$ sharpens for $a>1$
- **Limits:** $T\to0$: one-hot on $\arg\max$ (ties split evenly); $T\to\infty$: uniform $1/K$
- **Monotone:** order-preserving: $z_i>z_j\iff p_i>p_j$, so $\arg\max p=\arg\max z$
- **Gradient with CE:** $p-e_y$, bounded in $[-1,1]$ per coordinate, zero only when $p=e_y$
- **Cost:** $O(K)$ exponentials; for vocabularies of $10^5$ tokens the output softmax is a noticeable share of compute

## Intuition and history

The formula is the Boltzmann (Gibbs) distribution of statistical mechanics, $p_i\propto e^{-E_i/kT}$, with logits as negative energies; that is where "temperature" comes from. In statistics it is the multinomial logit model. The name *softmax* is due to John Bridle (1989–1990), who proposed it as the output of a neural classifier so that the outputs could be read as posterior probabilities.

"Soft max" is slightly misleading: the soft version of $\max$ is log-sum-exp. Softmax is the soft version of $\arg\max$ as a one-hot vector, and it is exactly the gradient of LSE: $\nabla_z\operatorname{LSE}(z)=\operatorname{softmax}(z)$.

Temperature divides the logits before exponentiating. $T<1$ exaggerates the gaps and sharpens the distribution, $T>1$ shrinks them and flattens it. Language-model sampling uses exactly this knob (see **Temperature, Top-k & Top-p**), and knowledge distillation uses $T>1$ so a teacher's small probabilities carry signal.

In attention, $\operatorname{softmax}(QK^\top/\sqrt{d_k})$ turns query–key dot products into mixing weights. If the components of $q$ and $k$ are independent with unit variance, $q\cdot k$ has variance $d_k$, so for $d_k=64$ the scores have standard deviation 8, which would push the softmax into near one-hot territory. Dividing by $\sqrt{d_k}=8$ restores unit variance. In a quick simulation (16 random keys, 2000 draws) the largest attention weight averaged 0.84 without the scaling and 0.24 with it.

> Figure: Drag the logits and watch every bar move at once. The Jacobian shows each logit's effect on each probability: positive diagonal, negative off-diagonal, rows summing to zero. Pick a target class to see the cross-entropy gradient p − y. Press +1 to every z: the logits change, the probabilities and gradients do not. Lower T to sharpen; the Jacobian fades as p approaches one-hot.

## Worked example: three logits

Take $z=(2.0,\,1.0,\,0.1)$ and true class $y=1$ (the first). Subtract $m=2$: exponents $(e^0,e^{-1},e^{-1.9})=(1,\,0.3679,\,0.1496)$, sum $1.5174$.

Forward and backward pass (4 decimals, computed in float64)

| Quantity | Computation | Value |
| --- | --- | --- |
| $p$ | $(1,\,0.3679,\,0.1496)/1.5174$ | (0.6590, 0.2424, 0.0986) |
| $\operatorname{LSE}(z)$ | $2+\log1.5174$, between $\max z=2$ and $2+\log3=3.0986$ | 2.4170 |
| $\mathcal L=-\log p_1$ | $\operatorname{LSE}(z)-z_1=2.4170-2$ | 0.4170 |
| $\partial\mathcal L/\partial z$ | $p-e_1$ | (−0.3410, 0.2424, 0.0986) |
| $J$ row 1 | $(p_1(1-p_1),\,-p_1p_2,\,-p_1p_3)$ | (0.2247, −0.1598, −0.0650) |
| $J$ row 2 | $(-p_2p_1,\,p_2(1-p_2),\,-p_2p_3)$ | (−0.1598, 0.1837, −0.0239) |
| $J$ row 3 | $(-p_3p_1,\,-p_3p_2,\,p_3(1-p_3))$ | (−0.0650, −0.0239, 0.0889) |
| $\operatorname{softmax}(z/0.5)$ | T = 0.5, sharper | (0.8638, 0.1169, 0.0193) |
| $\operatorname{softmax}(z/2)$ | T = 2, flatter | (0.5017, 0.3043, 0.1940) |
| $\operatorname{softmax}(z+100)$ | shift every logit | (0.6590, 0.2424, 0.0986) |

The gradient components sum to zero, like every column of $J$. The true class gets $-0.341$: gradient descent raises $z_1$ and lowers the other two in proportion to how much probability they stole. The entropy is 1.222 bits at $T=1$, 0.655 at $T=0.5$ and 1.481 at $T=2$ (the maximum for three classes is $\log_23=1.585$).

## Effect on training: saturation

Multiply the example's logits by 10, $z=(20,10,1)$, so $p_1=0.99995$. Every Jacobian entry is now tiny ($p_1(1-p_1)=4.5\times10^{-5}$): the softmax has **saturated**, and a small change of logits barely moves the probabilities.

Whether that hurts depends on the loss. Suppose the true class is the third one ($p_3=5.6\times10^{-9}$, a confident mistake). Cross-entropy gives $\mathcal L=19.0$ and gradient $p-e_3=(1.0,\,4.5\times10^{-5},\,-1.0)$: full strength, because the $\log$ cancels the exponential. Squared error on the probabilities instead gives $2J^\top(p-e_3)=(9.1\times10^{-5},\,-9.1\times10^{-5},\,-2.2\times10^{-8})$, about $10^4$ times weaker. This is why softmax is paired with cross-entropy (or log-likelihood) and almost never with MSE.

> **Never apply softmax before nn.CrossEntropyLoss:** `nn.CrossEntropyLoss` and `F.cross_entropy` expect raw logits and apply `log_softmax` internally. Feeding them probabilities applies softmax twice: the model still trains, but the effective outputs are squashed toward uniform and gradients shrink. Likewise use `log_softmax` (one fused, stable op) instead of `torch.log(torch.softmax(z))`, which returns `-inf` once a probability underflows to 0.

> **Overflow without the max trick:** $e^{z}$ overflows for $z>88.7$ in float32 and $z>709.8$ in float64 (and much sooner in float16). A naive softmax of `[1000., 999.]` computes `inf/inf = nan`; the shifted version gives $(0.7311,\,0.2689)$, which is $\sigma(1)$ and $1-\sigma(1)$.

> **Softmax only sees differences:** Because of shift invariance a $K$-class softmax has $K-1$ degrees of freedom. One logit can be fixed at 0 without losing anything (multinomial logistic regression does this). The redundancy is harmless for training but explains why the Jacobian is singular.

```python
import torch, torch.nn.functional as F

z = torch.tensor([2.0, 1.0, 0.1], requires_grad=True)
p = torch.softmax(z, dim=-1)                 # [0.6590, 0.2424, 0.0986]
loss = F.cross_entropy(z[None], torch.tensor([0]))   # 0.4170, takes raw logits
loss.backward()
print(z.grad)                                # [-0.3410, 0.2424, 0.0986] = p - onehot
print(torch.logsumexp(z, -1))                # 2.4170
print(torch.softmax(z / 0.5, -1))            # [0.8638, 0.1169, 0.0193]

def softmax(z, T=1.0, dim=-1):
    z = z / T
    z = z - z.amax(dim=dim, keepdim=True)    # shift invariance: largest exponent is e^0
    e = z.exp()
    return e / e.sum(dim=dim, keepdim=True)

def softmax_jacobian(p):                     # p: (K,)
    return torch.diag(p) - torch.outer(p, p)

J = softmax_jacobian(softmax(z.detach()))
print(J.sum(0))                              # [0., 0., 0.]
print(softmax(torch.tensor([1000.0, 999.0])))  # [0.7311, 0.2689], no overflow
```

Library calls, then a from-scratch stable softmax with its Jacobian

**Strengths**

- Outputs are a valid distribution, so they can be read as class probabilities and sampled from.
- Smooth and differentiable everywhere; with cross-entropy the gradient is the clean, bounded $p-y$.
- One temperature knob trades confidence for diversity without retraining.
- Stable and cheap to compute with the max trick; fused log-softmax kernels everywhere.

**Limitations**

- Never outputs exact zeros, so it cannot express "impossible"; sparse alternatives (sparsemax, entmax) can.
- Saturates for large logit gaps: the Jacobian vanishes, which hurts any loss other than log-likelihood.
- Probabilities are often over-confident and need calibration (e.g. temperature scaling on a validation set).
- $O(K)$ normalisation over huge vocabularies; attention softmax over $n$ keys is the $O(n^2)$ bottleneck.
- Classes compete: a single-label assumption. Multi-label problems need one sigmoid per label instead.

**Question:** You add 5 to every logit fed into a softmax, then separately multiply every logit by 5. What happens to the probabilities?

- A. Both leave them unchanged
- B. Adding leaves them unchanged; multiplying sharpens them toward argmax
- C. Adding sharpens them; multiplying leaves them unchanged
- D. Both flatten them toward uniform

**Answer:** B. Adding leaves them unchanged; multiplying sharpens them toward argmax — $e^{z_i+5}/\sum_j e^{z_j+5}=e^{z_i}/\sum_je^{z_j}$, so shifts cancel. Multiplying by 5 is the same as temperature $T=1/5$, which magnifies the gaps between logits and pushes the distribution toward one-hot.

**Question:** Logits $z=(2,1,0.1)$, true class 1, cross-entropy loss. What is $\partial\mathcal L/\partial z_2$?

- A. −0.2424
- B. 0
- C. 0.2424
- D. −0.6590

**Answer:** C. 0.2424 — $\partial\mathcal L/\partial z=p-e_y$. Class 2 is not the target, so its component is $+p_2=0.2424$: gradient descent lowers its logit. Only the true class gets a negative component, $p_1-1=-0.3410$.

**Key points**

- $p_i=e^{z_i}/\sum_je^{z_j}$: vector-valued, every output depends on every logit.
- Shift invariant, so subtract $\max z$ before exponentiating; $\log p=z-\operatorname{LSE}(z)$.
- Jacobian $\operatorname{diag}(p)-pp^\top$: symmetric, rows sum to 0, entries $\le0.25$, vanishing when saturated.
- With cross-entropy the gradient is $p-y$, which does not vanish on confident mistakes.
- Two classes reduce to $\sigma(z_1-z_2)$; temperature $T\to0$ gives argmax, $T\to\infty$ uniform.
- Attention divides scores by $\sqrt{d_k}$ to keep the softmax out of saturation.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
