Temperature, Top-k & Top-p

Temperature reshapes the next-token distribution; top-k and top-p truncate it before sampling.

3 min read134 wordscore2 viewsupdated 2026-10-03

An LLM outputs one logit ziz_i per vocabulary token. Sampling turns those logits into a choice, and three knobs control how adventurous that choice is.

pi=ezi/T∑jezj/Tp_i=\frac{e^{z_i/T}}{\sum_{j}e^{z_j/T}}
ziz_i
logit of token i
TT
temperature, T > 0
catz=2.0
0.638
dogz=1.0
0.235
foxz=0.1
0.095
carz=-1.0
0.032
H = 1.39 bits

T → 0: argmax (one-hot). T → ∞: uniform (H → 2 bits).

fig.Drag T. Entropy H measures how spread out the distribution is (at most 2 bits for 4 tokens).

Temperature rescales logits before the softmax: T<1T<1 sharpens, T>1T>1 flattens, and T→0T\to0 becomes greedy argmax.

Top-k keeps the kk most likely tokens and renormalizes. Top-p (nucleus) keeps the smallest set whose probabilities sum to at least pp, so the set shrinks when the model is confident and grows when it isn't.

Order of operations

  1. 1Divide logits by TT.
  2. 2Apply softmax.
  3. 3Truncate with top-k and/or top-p.
  4. 4Renormalize and sample.
python
1import numpy as np
2
3def sample(logits, T=1.0, top_p=0.9, rng=np.random.default_rng()):
4 z = (logits - logits.max()) / T
5 p = np.exp(z); p /= p.sum()
6 order = np.argsort(p)[::-1]
7 keep = order[: np.searchsorted(np.cumsum(p[order]), top_p) + 1]
8 q = np.zeros_like(p); q[keep] = p[keep]; q /= q.sum()
9 return rng.choice(len(p), p=q)

Check yourself

Logits [2, 1, 0.1, −1] at T = 0.1. The sampled token is almost always…

Remember

  • ✓T < 1 sharpens, T > 1 flattens.
  • ✓Top-p adapts the candidate set to the model's confidence.
  • ✓Always subtract the max logit.
← back to Generative AI roadmap