# Temperature, Top-k & Top-p

> Temperature reshapes the next-token distribution; top-k and top-p truncate it before sampling.

Source: https://www.mlfoundry.in/learn/temperature-top-k-top-p  
Section: Generative AI  
Level: core · 3 min read · updated 2026-10-03

An LLM outputs one logit $z_i$ per vocabulary token. Sampling turns those logits into a choice, and three knobs control how adventurous that choice is.

$$
p_i=\frac{e^{z_i/T}}{\sum_{j}e^{z_j/T}}
$$

- $z_i$: logit of token i
- $T$: temperature, T > 0

> Figure: Drag T. Entropy H measures how spread out the distribution is (at most 2 bits for 4 tokens).

**Temperature** rescales logits before the softmax: $T<1$ sharpens, $T>1$ flattens, and $T\to0$ becomes greedy argmax.

**Top-k** keeps the $k$ most likely tokens and renormalizes. **Top-p** (nucleus) keeps the smallest set whose probabilities sum to at least $p$, so the set shrinks when the model is confident and grows when it isn't.

**Order of operations**

1. Divide logits by $T$.
2. Apply softmax.
3. Truncate with top-k and/or top-p.
4. Renormalize and sample.

```python
import numpy as np

def sample(logits, T=1.0, top_p=0.9, rng=np.random.default_rng()):
    z = (logits - logits.max()) / T
    p = np.exp(z); p /= p.sum()
    order = np.argsort(p)[::-1]
    keep = order[: np.searchsorted(np.cumsum(p[order]), top_p) + 1]
    q = np.zeros_like(p); q[keep] = p[keep]; q /= q.sum()
    return rng.choice(len(p), p=q)
```

> **Overflow:** Subtract the max logit before exponentiating. $e^{1000}$ is `inf` in float64, and shifting every logit by a constant leaves the softmax unchanged.

**Question:** Logits [2, 1, 0.1, −1] at T = 0.1. The sampled token is almost always…

- A. uniform over all four
- B. the token with logit 2
- C. the token with logit −1
- D. impossible to predict

**Answer:** B. the token with logit 2 — Dividing by 0.1 makes the gap between the top two logits 10, so $p_1/p_2 = e^{10} \approx 22{,}000$.

**Key points**

- T < 1 sharpens, T > 1 flattens.
- Top-p adapts the candidate set to the model's confidence.
- Always subtract the max logit.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
