# Top-k and top-p sampling

> Top-k keeps the k most likely tokens; top-p (nucleus, Holtzman et al. 2020) keeps the smallest set whose probability reaches p. Both zero the tail and renormalise, but only top-p adapts the set size to the model's confidence.

Source: https://www.mlfoundry.in/learn/top-k-and-top-p-sampling  
Section: Generative AI › Decoding and Generation Control  
Level: core · 7 min read · updated 2026-10-21

**Top-k and top-p sampling:** are truncation rules applied to the next-token distribution before sampling. **Top-k** keeps only the $k$ highest-probability tokens; it was popularised for story generation by Fan, Lewis and Dauphin ("Hierarchical Neural Story Generation", ACL 2018) and used with $k=40$ for the GPT-2 samples (Radford et al., 2019). **Top-p**, or **nucleus sampling**, keeps the smallest set of top tokens whose cumulative probability is at least $p$; it was introduced by Holtzman, Buys, Du, Forbes and Choi in "The Curious Case of Neural Text Degeneration" (ICLR 2020). Both renormalise the survivors and sample from them.

Pure sampling from a language model has a tail problem. Each of the tens of thousands of unlikely tokens has a tiny probability, but together they hold several percent of the mass. Over a few hundred steps, at least one junk token is almost certain, and the text that follows conditions on it.

**Temperature sampling** cannot fix this: lowering $T$ shrinks the tail but also removes diversity among good tokens. Truncation removes the tail outright and leaves the head untouched. It runs after temperature and before the random draw; **Min-p sampling** is a third rule with the same role.

## The two rules

$$
V_k=\{\text{the }k\text{ tokens with largest }p_i\},\qquad V_p=\arg\min_{V'}\;|V'|\ \ \text{s.t.}\ \sum_{i\in V'}p_i\ge p
$$

- $p_i$: next-token probability after temperature
- $V_k$: top-k set: fixed size, variable mass
- $V_p$: nucleus: fixed mass, variable size; built by adding tokens in descending order until the running sum reaches p

$$
q_i=\frac{p_i\,\mathbb 1[i\in V]}{\sum_{j\in V}p_j}
$$

- $V$: the kept set, $V_k$, $V_p$, or both applied in sequence
- $q_i$: renormalised probability actually sampled from; every cut token gets exactly 0

Renormalisation scales every survivor by the same factor $1/\sum_{j\in V}p_j$, so ratios among kept tokens are unchanged. In logit space, implementations set cut logits to $-\infty$ and let softmax do the rest.

> Figure: The same two rules on two contexts. Flat ("My favourite food is"): top token 0.129; k = 5 keeps only 0.517 of the mass and cuts 8 plausible foods, while the p = 0.9 nucleus holds 13 tokens (0.912), all foods. Peaked ("The capital of France is"): Paris has 0.790; k = 5 still keeps five tokens, while the nucleus stops at four (cumulative 0.903).

## Why a fixed k fails

Holtzman et al. argued that the right number of candidates changes from step to step. After "My favourite food is", dozens of continuations are fine, and a small $k$ cuts good ones. After "The capital of France is", one token dominates, and a large $k$ admits dozens of poor ones. Top-p adapts: the nucleus is small when the model is confident and large when it is not.

Top-p has its own weak spot. When the distribution is flat, reaching a high $p$ can require dipping into the tail, and the set size jumps. In the figure below, raising $p$ from 0.90 to 0.95 on the flat context grows the nucleus from 13 tokens to 384. That sensitivity, worse at high temperature, motivated **Min-p sampling**.

**Settings worth knowing (documented behaviour as of 2024–2025)**

- **Hugging Face transformers:** `top_k=50` and `top_p=1.0` by default, active only with `do_sample=True`; the warpers run temperature, then top-k, then top-p, and always keep at least one token
- **OpenAI Chat Completions:** `top_p` (default 1), no `top_k`; the docs recommend changing `temperature` or `top_p`, not both
- **Anthropic Messages API:** exposes both `top_k` and `top_p`; the docs suggest most users only need `temperature`
- **Holtzman et al. (2020):** compared greedy, beam, pure, top-k and nucleus sampling on GPT-2 Large continuations; nucleus with $p=0.95$ came closest to human text on perplexity and repetition
- **Order matters:** top-p after top-k works on the renormalised top-k distribution, so the combined set can be smaller than either rule alone (worked example below)

> Figure: Two toy contexts, each with 16 named tokens plus 1,000 junk tail tokens. Default (flat context, top-p = 0.9): 13 of 1,016 tokens kept, 0.912 of the mass, all 13 foods kept, junk mass 0, P(junk in 50 steps) = 0. With no truncation the junk mass is 0.0882 and P(junk in 50 steps) = 0.990. Raise p to 0.95: 384 tokens kept, junk 0.0402, P = 0.872. Switch to the peaked context: p = 0.9 keeps 4 tokens; top-k = 50 keeps 34 tail tokens, junk 0.0082, P = 0.338; top-k 50 → top-p 0.9 keeps just 3.

## Worked example

Take a peaked distribution: Paris 0.790, located 0.048, a 0.036, the 0.029, home 0.018, then 79 tail tokens at 0.001 each (total 1).

Nucleus with p = 0.9: add tokens until the running sum reaches 0.9, then divide by the kept mass 0.903

| Rank | Token | p | Cumulative | In V₀.₉? | q = p / 0.903 |
| --- | --- | --- | --- | --- | --- |
| 1 | Paris | 0.790 | 0.790 | yes | 0.875 |
| 2 | located | 0.048 | 0.838 | yes | 0.053 |
| 3 | a | 0.036 | 0.874 | yes | 0.040 |
| 4 | the | 0.029 | 0.903 | yes (reaches 0.9) | 0.032 |
| 5 | home | 0.018 | 0.921 | no | 0 |
| 6–84 | tail × 79 | 0.001 each | 1.000 | no | 0 |

**The same distribution under other settings**

1. **Top-k = 2**: keep Paris and located, mass 0.838; $q=0.790/0.838=0.943$ and $0.057$.
2. **Top-k = 50**: keep 5 named tokens and 45 tail tokens, mass 0.966. The tail still holds $0.045/0.966=4.7\%$ of the sampling mass.
3. **Top-k = 50, then top-p = 0.9**: on the renormalised top-50 the running sums are 0.818, 0.868, 0.905, so only **3** tokens survive, one fewer than top-p alone.

```python
import numpy as np

def truncate(probs, top_k=0, top_p=1.0):
    """Hugging Face order: top-k first, then top-p on the renormalised survivors."""
    p = np.asarray(probs, dtype=np.float64)
    order = np.argsort(-p, kind="stable")
    keep = np.zeros(p.size, dtype=bool)
    n = p.size if top_k <= 0 else min(top_k, p.size)
    keep[order[:n]] = True
    if top_p < 1.0:
        q = p[order[:n]] / p[order[:n]].sum()
        m = int(np.searchsorted(np.cumsum(q), top_p) + 1)   # smallest prefix with mass >= top_p
        keep[:] = False
        keep[order[:m]] = True
    out = np.where(keep, p, 0.0)
    return out / out.sum()

# Paris, located, a, the, home, then a tail of 79 tokens at 0.001 each (sums to 1)
p = np.r_[0.790, 0.048, 0.036, 0.029, 0.018, np.full(79, 0.001)]
print(truncate(p, top_p=0.9)[:6].round(3))     # [0.875 0.053 0.04  0.032 0.    0.   ]
print(truncate(p, top_k=2)[:6].round(3))       # [0.943 0.057 0.    0.    0.    0.   ]
print((truncate(p, top_k=50) > 0).sum(),       # 50
      (truncate(p, top_k=50, top_p=0.9) > 0).sum())   # 3
```

Top-k then top-p in the Hugging Face order (numpy; runs as is). In transformers the same thing is generate(..., do_sample=True, top_k=50, top_p=0.9)

**Strengths**

- Top-k: one integer, trivial to implement, hard cap on candidates; cheap on GPU with a partial sort.
- Top-p: set size follows the model's confidence, so one value works across flat and peaked steps.
- Both remove the tail exactly, which temperature alone cannot do.

**Limitations**

- Top-k: cuts good tokens on flat steps and admits junk on peaked ones.
- Top-p: needs a full sort and cumulative sum per step; at high p on flat steps the nucleus can swell into the tail.
- Both interact with temperature and with each other, so tuning one changes the effect of the others.

> **p = 1 and k = 0 mean off:** `top_p=1.0` keeps everything, and in most libraries `top_k=0` (or `-1`) disables top-k. Very small values are the opposite extreme: `top_k=1` or a `top_p` below the top token's probability is greedy decoding, whatever the temperature.

> **Truncation and temperature together:** With truncation in place, a moderately high temperature becomes safer: it spreads mass among the kept tokens but cannot reach the cut tail. Because temperature runs first in common stacks, a higher $T$ also flattens $p$ and so enlarges the nucleus for a fixed $p$ (see **Temperature sampling**).

**Question:** The top token has probability 0.93. Which tokens does top-p = 0.9 keep?

- A. The top 90% of the vocabulary
- B. Only the top token
- C. The top 50 tokens
- D. Every token with probability ≥ 0.1

**Answer:** B. Only the top token — The nucleus is the smallest prefix whose cumulative mass reaches $p$. One token already reaches 0.93 ≥ 0.9, so the step becomes greedy.

**Question:** Why can top-k = 50 followed by top-p = 0.9 keep fewer tokens than top-p = 0.9 alone?

- A. Top-k reorders the tokens
- B. Top-p then measures mass on the renormalised top-50 distribution, where each kept token's share is larger, so 0.9 is reached sooner
- C. Top-k raises the temperature
- D. It cannot; the combined set is always the larger of the two

**Answer:** B. Top-p then measures mass on the renormalised top-50 distribution, where each kept token's share is larger, so 0.9 is reached sooner — Dropping the tail raises every survivor's share by $1/\sum_{V_k}p_j$. In the worked example that factor is $1/0.966$, so the running sums 0.790, 0.838, 0.874 become 0.818, 0.868, 0.905, and the 0.9 threshold falls at rank 3 instead of 4.

**Key points**

- Top-k keeps a fixed count; top-p keeps the smallest prefix with mass ≥ p; both renormalise and sample.
- A fixed k is too tight on flat steps and too loose on peaked ones; the nucleus adapts its size.
- Top-p can still swell into the tail at high p or high T; Min-p scales the cut-off with the top token instead.
- Hugging Face: temperature → top-k → top-p, defaults top_k = 50 and top_p = 1.0 with do_sample=True.
- top_k = 1 or a tiny top_p is greedy; top_p = 1 and top_k = 0 switch truncation off.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
