Softmax

Exponentiate and normalise: turns a vector of logits into a probability distribution whose cross-entropy gradient is simply p − y.

9 min read847 wordscore0 viewsupdated 2026-10-11

Definition

Softmax maps a vector of real scores (logits) z∈RKz\in\mathbb{R}^K to a probability vector p∈ΔK−1p\in\Delta^{K-1} by exponentiating each score and dividing by the sum: pi=ezi/∑jezjp_i=e^{z_i}/\sum_j e^{z_j}. Every pip_i is positive and they sum to 1. It is a smooth, differentiable stand-in for arg⁡max⁡\arg\max.

Most activations act on one number at a time. Softmax does not: each output depends on every input, because they all share the denominator. Raise one logit and its probability goes up while all the others go down. This coupling is what makes it a distribution, and it is why its derivative is a matrix, not a scalar.

It sits at the end of almost every classifier (one logit per class) and inside every attention layer (one score per key). The output layer of a language model is a softmax over the whole vocabulary.

Formula

softmax⁡(z)i=ezi/T∑j=1Kezj/T,log⁡softmax⁡(z)i=ziT−LSE⁡ ⁣(zT)\operatorname{softmax}(z)_i=\frac{e^{z_i/T}}{\sum_{j=1}^{K}e^{z_j/T}},\qquad \log\operatorname{softmax}(z)_i=\frac{z_i}{T}-\operatorname{LSE}\!\left(\frac{z}{T}\right)
ziz_i
logit (unnormalised score) of class i
KK
number of classes, keys or vocabulary tokens
TT
temperature, T>0T>0; T=1T=1 is the plain softmax
LSE⁡(z)\operatorname{LSE}(z)
log-sum-exp log⁡∑jezj\log\sum_j e^{z_j}, a smooth maximum: max⁡jzj≤LSE⁡(z)≤max⁡jzj+log⁡K\max_j z_j\le\operatorname{LSE}(z)\le\max_j z_j+\log K

Shift invariance. Adding the same constant cc to every logit multiplies numerator and denominator by ece^{c}, so nothing changes: softmax⁡(z+c1)=softmax⁡(z)\operatorname{softmax}(z+c\mathbf 1)=\operatorname{softmax}(z). Only differences between logits matter. Every implementation uses this to subtract m=max⁡jzjm=\max_j z_j first, so the largest exponent is e0=1e^0=1 and nothing can overflow:

pi=ezi−m∑jezj−m,LSE⁡(z)=m+log⁡∑jezj−mp_i=\frac{e^{z_i-m}}{\sum_j e^{z_j-m}},\qquad \operatorname{LSE}(z)=m+\log\sum_j e^{z_j-m}
mm
max⁡jzj\max_j z_j; any constant works mathematically, the max is the one that keeps every exponent ≤0\le0

Two classes is a sigmoid. With logits (z1,z2)(z_1,z_2), p1=1/(1+e−(z1−z2))=σ(z1−z2)p_1=1/(1+e^{-(z_1-z_2)})=\sigma(z_1-z_2). Fix z2=0z_2=0 and the softmax is exactly the logistic sigmoid, e.g. softmax⁡(1.5,0)1=σ(1.5)=0.8176\operatorname{softmax}(1.5,0)_1=\sigma(1.5)=0.8176. Softmax is the multi-class generalisation of logistic regression's output.

Derivative: the Jacobian

Differentiate pip_i with respect to each logit zjz_j. The diagonal term comes from the numerator, the off-diagonal ones from the shared denominator:

∂pi∂zj=1T pi(δij−pj),J=1T(diag⁡(p)−p p⊤)\frac{\partial p_i}{\partial z_j}=\frac{1}{T}\,p_i\big(\delta_{ij}-p_j\big),\qquad J=\frac{1}{T}\Big(\operatorname{diag}(p)-p\,p^{\top}\Big)
δij\delta_{ij}
Kronecker delta: 1 if i=ji=j, else 0
pi(1−pi)p_i(1-p_i)
diagonal entries, positive, at most 0.250.25 (at pi=0.5p_i=0.5), the sigmoid's slope again
−pipj-p_ip_j
off-diagonal entries, always negative: raising one logit lowers every other probability

JJ is symmetric and positive semidefinite, and every row and column sums to zero, because ∑ipi=1\sum_i p_i=1 cannot change. Its null space is the direction 1\mathbf 1: the matrix form of shift invariance.

In practice you never build JJ. Softmax is followed by cross-entropy L=−log⁡py\mathcal L=-\log p_y, and the product J⊤∇pLJ^\top\nabla_p\mathcal L collapses to something very simple:

L=−log⁡py=LSE⁡(z)−zy⟹∂L∂z=p−ey(T=1)\mathcal L=-\log p_y=\operatorname{LSE}(z)-z_y\quad\Longrightarrow\quad \frac{\partial\mathcal L}{\partial z}=p-e_y\quad(T=1)
eye_y
one-hot vector of the true class yy
p−eyp-e_y
predicted minus target: py−1<0p_y-1<0 for the true class (push its logit up), +pi+p_i for every other class (push down)

Properties

Input → output
RK→\mathbb{R}^K\to the open probability simplex: every pi∈(0,1)p_i\in(0,1), ∑ipi=1\sum_i p_i=1
Not elementwise
each pip_i depends on all zjz_j; the derivative is a K×KK\times K Jacobian
Invariance
softmax⁡(z+c1)=softmax⁡(z)\operatorname{softmax}(z+c\mathbf 1)=\operatorname{softmax}(z); scaling is not invariant: softmax⁡(az)\operatorname{softmax}(az) sharpens for a>1a>1
Limits
T→0T\to0: one-hot on arg⁡max⁡\arg\max (ties split evenly); T→∞T\to\infty: uniform 1/K1/K
Monotone
order-preserving: zi>zj  ⟺  pi>pjz_i>z_j\iff p_i>p_j, so arg⁡max⁡p=arg⁡max⁡z\arg\max p=\arg\max z
Gradient with CE
p−eyp-e_y, bounded in [−1,1][-1,1] per coordinate, zero only when p=eyp=e_y
Cost
O(K)O(K) exponentials; for vocabularies of 10510^5 tokens the output softmax is a noticeable share of compute

Intuition and history

The formula is the Boltzmann (Gibbs) distribution of statistical mechanics, pi∝e−Ei/kTp_i\propto e^{-E_i/kT}, with logits as negative energies; that is where "temperature" comes from. In statistics it is the multinomial logit model. The name softmax is due to John Bridle (1989–1990), who proposed it as the output of a neural classifier so that the outputs could be read as posterior probabilities.

"Soft max" is slightly misleading: the soft version of max⁡\max is log-sum-exp. Softmax is the soft version of arg⁡max⁡\arg\max as a one-hot vector, and it is exactly the gradient of LSE: ∇zLSE⁡(z)=softmax⁡(z)\nabla_z\operatorname{LSE}(z)=\operatorname{softmax}(z).

Temperature divides the logits before exponentiating. T<1T<1 exaggerates the gaps and sharpens the distribution, T>1T>1 shrinks them and flattens it. Language-model sampling uses exactly this knob (see Temperature, Top-k & Top-p), and knowledge distillation uses T>1T>1 so a teacher's small probabilities carry signal.

In attention, softmax⁡(QK⊤/dk)\operatorname{softmax}(QK^\top/\sqrt{d_k}) turns query–key dot products into mixing weights. If the components of qq and kk are independent with unit variance, q⋅kq\cdot k has variance dkd_k, so for dk=64d_k=64 the scores have standard deviation 8, which would push the softmax into near one-hot territory. Dividing by dk=8\sqrt{d_k}=8 restores unit variance. In a quick simulation (16 random keys, 2000 draws) the largest attention weight averaged 0.84 without the scaling and 0.24 with it.

p = softmax(z / T), T = 1.00probability pᵢ∂L/∂zᵢ = (pᵢ − yᵢ)/Tcat ★z=+2.000.614−0.386dogz=+1.000.2260.226foxz=+0.100.0920.092owlz=−0.500.0500.050carz=−1.500.0190.019Jacobian ∂pᵢ/∂zⱼ= (diag p − ppᵀ)/Tcatdogfoxowlcarcat.24−.14−.06−.03−.01dog−.14.17−.02−.010fox−.06−.02.0800owl−.03−.010.050car−.01000.02■ + ■ − · solid at |J| ≥ 0.25rows and columns sum to 0LSE = log Σ e^(zⱼ/T) = 2.488 · max zⱼ/T = 2.000 · entropy 1.557 bits (max 2.32)target ★ cat: L = −log p = 0.488 · green pushes zₖ up, rose pushes the rest downadd a constant to every logit to see shift invariance
target y
all logits
fig.Drag the logits and watch every bar move at once. The Jacobian shows each logit's effect on each probability: positive diagonal, negative off-diagonal, rows summing to zero. Pick a target class to see the cross-entropy gradient p − y. Press +1 to every z: the logits change, the probabilities and gradients do not. Lower T to sharpen; the Jacobian fades as p approaches one-hot.

Worked example: three logits

Take z=(2.0, 1.0, 0.1)z=(2.0,\,1.0,\,0.1) and true class y=1y=1 (the first). Subtract m=2m=2: exponents (e0,e−1,e−1.9)=(1, 0.3679, 0.1496)(e^0,e^{-1},e^{-1.9})=(1,\,0.3679,\,0.1496), sum 1.51741.5174.

QuantityComputationValue
pp(1, 0.3679, 0.1496)/1.5174(1,\,0.3679,\,0.1496)/1.5174(0.6590, 0.2424, 0.0986)
LSE⁡(z)\operatorname{LSE}(z)2+log⁡1.51742+\log1.5174, between max⁡z=2\max z=2 and 2+log⁡3=3.09862+\log3=3.09862.4170
L=−log⁡p1\mathcal L=-\log p_1LSE⁡(z)−z1=2.4170−2\operatorname{LSE}(z)-z_1=2.4170-20.4170
∂L/∂z\partial\mathcal L/\partial zp−e1p-e_1(−0.3410, 0.2424, 0.0986)
JJ row 1(p1(1−p1), −p1p2, −p1p3)(p_1(1-p_1),\,-p_1p_2,\,-p_1p_3)(0.2247, −0.1598, −0.0650)
JJ row 2(−p2p1, p2(1−p2), −p2p3)(-p_2p_1,\,p_2(1-p_2),\,-p_2p_3)(−0.1598, 0.1837, −0.0239)
JJ row 3(−p3p1, −p3p2, p3(1−p3))(-p_3p_1,\,-p_3p_2,\,p_3(1-p_3))(−0.0650, −0.0239, 0.0889)
softmax⁡(z/0.5)\operatorname{softmax}(z/0.5)T = 0.5, sharper(0.8638, 0.1169, 0.0193)
softmax⁡(z/2)\operatorname{softmax}(z/2)T = 2, flatter(0.5017, 0.3043, 0.1940)
softmax⁡(z+100)\operatorname{softmax}(z+100)shift every logit(0.6590, 0.2424, 0.0986)
Forward and backward pass (4 decimals, computed in float64)

The gradient components sum to zero, like every column of JJ. The true class gets −0.341-0.341: gradient descent raises z1z_1 and lowers the other two in proportion to how much probability they stole. The entropy is 1.222 bits at T=1T=1, 0.655 at T=0.5T=0.5 and 1.481 at T=2T=2 (the maximum for three classes is log⁡23=1.585\log_23=1.585).

Effect on training: saturation

Multiply the example's logits by 10, z=(20,10,1)z=(20,10,1), so p1=0.99995p_1=0.99995. Every Jacobian entry is now tiny (p1(1−p1)=4.5×10−5p_1(1-p_1)=4.5\times10^{-5}): the softmax has saturated, and a small change of logits barely moves the probabilities.

Whether that hurts depends on the loss. Suppose the true class is the third one (p3=5.6×10−9p_3=5.6\times10^{-9}, a confident mistake). Cross-entropy gives L=19.0\mathcal L=19.0 and gradient p−e3=(1.0, 4.5×10−5, −1.0)p-e_3=(1.0,\,4.5\times10^{-5},\,-1.0): full strength, because the log⁡\log cancels the exponential. Squared error on the probabilities instead gives 2J⊤(p−e3)=(9.1×10−5, −9.1×10−5, −2.2×10−8)2J^\top(p-e_3)=(9.1\times10^{-5},\,-9.1\times10^{-5},\,-2.2\times10^{-8}), about 10410^4 times weaker. This is why softmax is paired with cross-entropy (or log-likelihood) and almost never with MSE.

python
1import torch, torch.nn.functional as F
2
3z = torch.tensor([2.0, 1.0, 0.1], requires_grad=True)
4p = torch.softmax(z, dim=-1) # [0.6590, 0.2424, 0.0986]
5loss = F.cross_entropy(z[None], torch.tensor([0])) # 0.4170, takes raw logits
6loss.backward()
7print(z.grad) # [-0.3410, 0.2424, 0.0986] = p - onehot
8print(torch.logsumexp(z, -1)) # 2.4170
9print(torch.softmax(z / 0.5, -1)) # [0.8638, 0.1169, 0.0193]
10
11def softmax(z, T=1.0, dim=-1):
12 z = z / T
13 z = z - z.amax(dim=dim, keepdim=True) # shift invariance: largest exponent is e^0
14 e = z.exp()
15 return e / e.sum(dim=dim, keepdim=True)
16
17def softmax_jacobian(p): # p: (K,)
18 return torch.diag(p) - torch.outer(p, p)
19
20J = softmax_jacobian(softmax(z.detach()))
21print(J.sum(0)) # [0., 0., 0.]
22print(softmax(torch.tensor([1000.0, 999.0]))) # [0.7311, 0.2689], no overflow
Library calls, then a from-scratch stable softmax with its Jacobian

Use when

  • +Outputs are a valid distribution, so they can be read as class probabilities and sampled from.
  • +Smooth and differentiable everywhere; with cross-entropy the gradient is the clean, bounded p−yp-y.
  • +One temperature knob trades confidence for diversity without retraining.
  • +Stable and cheap to compute with the max trick; fused log-softmax kernels everywhere.

Avoid when

  • −Never outputs exact zeros, so it cannot express "impossible"; sparse alternatives (sparsemax, entmax) can.
  • −Saturates for large logit gaps: the Jacobian vanishes, which hurts any loss other than log-likelihood.
  • −Probabilities are often over-confident and need calibration (e.g. temperature scaling on a validation set).
  • −O(K)O(K) normalisation over huge vocabularies; attention softmax over nn keys is the O(n2)O(n^2) bottleneck.
  • −Classes compete: a single-label assumption. Multi-label problems need one sigmoid per label instead.

Check yourself

You add 5 to every logit fed into a softmax, then separately multiply every logit by 5. What happens to the probabilities?

Check yourself

Logits z=(2,1,0.1)z=(2,1,0.1), true class 1, cross-entropy loss. What is ∂L/∂z2\partial\mathcal L/\partial z_2?

Remember

  • ✓pi=ezi/∑jezjp_i=e^{z_i}/\sum_je^{z_j}: vector-valued, every output depends on every logit.
  • ✓Shift invariant, so subtract max⁡z\max z before exponentiating; log⁡p=z−LSE⁡(z)\log p=z-\operatorname{LSE}(z).
  • ✓Jacobian diag⁡(p)−pp⊤\operatorname{diag}(p)-pp^\top: symmetric, rows sum to 0, entries ≤0.25\le0.25, vanishing when saturated.
  • ✓With cross-entropy the gradient is p−yp-y, which does not vanish on confident mistakes.
  • ✓Two classes reduce to σ(z1−z2)\sigma(z_1-z_2); temperature T→0T\to0 gives argmax, T→∞T\to\infty uniform.
  • ✓Attention divides scores by dk\sqrt{d_k} to keep the softmax out of saturation.
← back to Deep Learning roadmap