An LLM outputs one logit per vocabulary token. Sampling turns those logits into a choice, and three knobs control how adventurous that choice is.
- logit of token i
- temperature, T > 0
catz=2.0
0.638
dogz=1.0
0.235
foxz=0.1
0.095
carz=-1.0
0.032
H = 1.39 bits
T → 0: argmax (one-hot). T → ∞: uniform (H → 2 bits).
Temperature rescales logits before the softmax: sharpens, flattens, and becomes greedy argmax.
Top-k keeps the most likely tokens and renormalizes. Top-p (nucleus) keeps the smallest set whose probabilities sum to at least , so the set shrinks when the model is confident and grows when it isn't.
Order of operations
- 1Divide logits by .
- 2Apply softmax.
- 3Truncate with top-k and/or top-p.
- 4Renormalize and sample.
python
1import numpy as np23def sample(logits, T=1.0, top_p=0.9, rng=np.random.default_rng()):4 z = (logits - logits.max()) / T5 p = np.exp(z); p /= p.sum()6 order = np.argsort(p)[::-1]7 keep = order[: np.searchsorted(np.cumsum(p[order]), top_p) + 1]8 q = np.zeros_like(p); q[keep] = p[keep]; q /= q.sum()9 return rng.choice(len(p), p=q)Check yourself
Logits [2, 1, 0.1, −1] at T = 0.1. The sampled token is almost always…
Remember
- ✓T < 1 sharpens, T > 1 flattens.
- ✓Top-p adapts the candidate set to the model's confidence.
- ✓Always subtract the max logit.