Definition
Softmax maps a vector of real scores (logits) to a probability vector by exponentiating each score and dividing by the sum: . Every is positive and they sum to 1. It is a smooth, differentiable stand-in for .
Most activations act on one number at a time. Softmax does not: each output depends on every input, because they all share the denominator. Raise one logit and its probability goes up while all the others go down. This coupling is what makes it a distribution, and it is why its derivative is a matrix, not a scalar.
It sits at the end of almost every classifier (one logit per class) and inside every attention layer (one score per key). The output layer of a language model is a softmax over the whole vocabulary.
Formula
- logit (unnormalised score) of class i
- number of classes, keys or vocabulary tokens
- temperature, ; is the plain softmax
- log-sum-exp , a smooth maximum:
Shift invariance. Adding the same constant to every logit multiplies numerator and denominator by , so nothing changes: . Only differences between logits matter. Every implementation uses this to subtract first, so the largest exponent is and nothing can overflow:
- ; any constant works mathematically, the max is the one that keeps every exponent
Two classes is a sigmoid. With logits , . Fix and the softmax is exactly the logistic sigmoid, e.g. . Softmax is the multi-class generalisation of logistic regression's output.
Derivative: the Jacobian
Differentiate with respect to each logit . The diagonal term comes from the numerator, the off-diagonal ones from the shared denominator:
- Kronecker delta: 1 if , else 0
- diagonal entries, positive, at most (at ), the sigmoid's slope again
- off-diagonal entries, always negative: raising one logit lowers every other probability
is symmetric and positive semidefinite, and every row and column sums to zero, because cannot change. Its null space is the direction : the matrix form of shift invariance.
In practice you never build . Softmax is followed by cross-entropy , and the product collapses to something very simple:
- one-hot vector of the true class
- predicted minus target: for the true class (push its logit up), for every other class (push down)
Properties
- Input → output
- the open probability simplex: every ,
- Not elementwise
- each depends on all ; the derivative is a Jacobian
- Invariance
- ; scaling is not invariant: sharpens for
- Limits
- : one-hot on (ties split evenly); : uniform
- Monotone
- order-preserving: , so
- Gradient with CE
- , bounded in per coordinate, zero only when
- Cost
- exponentials; for vocabularies of tokens the output softmax is a noticeable share of compute
Intuition and history
The formula is the Boltzmann (Gibbs) distribution of statistical mechanics, , with logits as negative energies; that is where "temperature" comes from. In statistics it is the multinomial logit model. The name softmax is due to John Bridle (1989–1990), who proposed it as the output of a neural classifier so that the outputs could be read as posterior probabilities.
"Soft max" is slightly misleading: the soft version of is log-sum-exp. Softmax is the soft version of as a one-hot vector, and it is exactly the gradient of LSE: .
Temperature divides the logits before exponentiating. exaggerates the gaps and sharpens the distribution, shrinks them and flattens it. Language-model sampling uses exactly this knob (see Temperature, Top-k & Top-p), and knowledge distillation uses so a teacher's small probabilities carry signal.
In attention, turns query–key dot products into mixing weights. If the components of and are independent with unit variance, has variance , so for the scores have standard deviation 8, which would push the softmax into near one-hot territory. Dividing by restores unit variance. In a quick simulation (16 random keys, 2000 draws) the largest attention weight averaged 0.84 without the scaling and 0.24 with it.
Worked example: three logits
Take and true class (the first). Subtract : exponents , sum .
| Quantity | Computation | Value |
|---|---|---|
| (0.6590, 0.2424, 0.0986) | ||
| , between and | 2.4170 | |
| 0.4170 | ||
| (−0.3410, 0.2424, 0.0986) | ||
| row 1 | (0.2247, −0.1598, −0.0650) | |
| row 2 | (−0.1598, 0.1837, −0.0239) | |
| row 3 | (−0.0650, −0.0239, 0.0889) | |
| T = 0.5, sharper | (0.8638, 0.1169, 0.0193) | |
| T = 2, flatter | (0.5017, 0.3043, 0.1940) | |
| shift every logit | (0.6590, 0.2424, 0.0986) |
The gradient components sum to zero, like every column of . The true class gets : gradient descent raises and lowers the other two in proportion to how much probability they stole. The entropy is 1.222 bits at , 0.655 at and 1.481 at (the maximum for three classes is ).
Effect on training: saturation
Multiply the example's logits by 10, , so . Every Jacobian entry is now tiny (): the softmax has saturated, and a small change of logits barely moves the probabilities.
Whether that hurts depends on the loss. Suppose the true class is the third one (, a confident mistake). Cross-entropy gives and gradient : full strength, because the cancels the exponential. Squared error on the probabilities instead gives , about times weaker. This is why softmax is paired with cross-entropy (or log-likelihood) and almost never with MSE.
1import torch, torch.nn.functional as F23z = torch.tensor([2.0, 1.0, 0.1], requires_grad=True)4p = torch.softmax(z, dim=-1) # [0.6590, 0.2424, 0.0986]5loss = F.cross_entropy(z[None], torch.tensor([0])) # 0.4170, takes raw logits6loss.backward()7print(z.grad) # [-0.3410, 0.2424, 0.0986] = p - onehot8print(torch.logsumexp(z, -1)) # 2.41709print(torch.softmax(z / 0.5, -1)) # [0.8638, 0.1169, 0.0193]1011def softmax(z, T=1.0, dim=-1):12 z = z / T13 z = z - z.amax(dim=dim, keepdim=True) # shift invariance: largest exponent is e^014 e = z.exp()15 return e / e.sum(dim=dim, keepdim=True)1617def softmax_jacobian(p): # p: (K,)18 return torch.diag(p) - torch.outer(p, p)1920J = softmax_jacobian(softmax(z.detach()))21print(J.sum(0)) # [0., 0., 0.]22print(softmax(torch.tensor([1000.0, 999.0]))) # [0.7311, 0.2689], no overflowUse when
- +Outputs are a valid distribution, so they can be read as class probabilities and sampled from.
- +Smooth and differentiable everywhere; with cross-entropy the gradient is the clean, bounded .
- +One temperature knob trades confidence for diversity without retraining.
- +Stable and cheap to compute with the max trick; fused log-softmax kernels everywhere.
Avoid when
- −Never outputs exact zeros, so it cannot express "impossible"; sparse alternatives (sparsemax, entmax) can.
- −Saturates for large logit gaps: the Jacobian vanishes, which hurts any loss other than log-likelihood.
- −Probabilities are often over-confident and need calibration (e.g. temperature scaling on a validation set).
- − normalisation over huge vocabularies; attention softmax over keys is the bottleneck.
- −Classes compete: a single-label assumption. Multi-label problems need one sigmoid per label instead.
Check yourself
You add 5 to every logit fed into a softmax, then separately multiply every logit by 5. What happens to the probabilities?
Check yourself
Logits , true class 1, cross-entropy loss. What is ?
Remember
- ✓: vector-valued, every output depends on every logit.
- ✓Shift invariant, so subtract before exponentiating; .
- ✓Jacobian : symmetric, rows sum to 0, entries , vanishing when saturated.
- ✓With cross-entropy the gradient is , which does not vanish on confident mistakes.
- ✓Two classes reduce to ; temperature gives argmax, uniform.
- ✓Attention divides scores by to keep the softmax out of saturation.