Definition
Top-k and top-p sampling are truncation rules applied to the next-token distribution before sampling. Top-k keeps only the highest-probability tokens; it was popularised for story generation by Fan, Lewis and Dauphin ("Hierarchical Neural Story Generation", ACL 2018) and used with for the GPT-2 samples (Radford et al., 2019). Top-p, or nucleus sampling, keeps the smallest set of top tokens whose cumulative probability is at least ; it was introduced by Holtzman, Buys, Du, Forbes and Choi in "The Curious Case of Neural Text Degeneration" (ICLR 2020). Both renormalise the survivors and sample from them.
Pure sampling from a language model has a tail problem. Each of the tens of thousands of unlikely tokens has a tiny probability, but together they hold several percent of the mass. Over a few hundred steps, at least one junk token is almost certain, and the text that follows conditions on it.
Temperature sampling cannot fix this: lowering shrinks the tail but also removes diversity among good tokens. Truncation removes the tail outright and leaves the head untouched. It runs after temperature and before the random draw; Min-p sampling is a third rule with the same role.
The two rules
- next-token probability after temperature
- top-k set: fixed size, variable mass
- nucleus: fixed mass, variable size; built by adding tokens in descending order until the running sum reaches p
- the kept set, , , or both applied in sequence
- renormalised probability actually sampled from; every cut token gets exactly 0
Renormalisation scales every survivor by the same factor , so ratios among kept tokens are unchanged. In logit space, implementations set cut logits to and let softmax do the rest.
Why a fixed k fails
Holtzman et al. argued that the right number of candidates changes from step to step. After "My favourite food is", dozens of continuations are fine, and a small cuts good ones. After "The capital of France is", one token dominates, and a large admits dozens of poor ones. Top-p adapts: the nucleus is small when the model is confident and large when it is not.
Top-p has its own weak spot. When the distribution is flat, reaching a high can require dipping into the tail, and the set size jumps. In the figure below, raising from 0.90 to 0.95 on the flat context grows the nucleus from 13 tokens to 384. That sensitivity, worse at high temperature, motivated Min-p sampling.
Settings worth knowing (documented behaviour as of 2024–2025)
- Hugging Face transformers
top_k=50andtop_p=1.0by default, active only withdo_sample=True; the warpers run temperature, then top-k, then top-p, and always keep at least one token- OpenAI Chat Completions
top_p(default 1), notop_k; the docs recommend changingtemperatureortop_p, not both- Anthropic Messages API
- exposes both
top_kandtop_p; the docs suggest most users only needtemperature - Holtzman et al. (2020)
- compared greedy, beam, pure, top-k and nucleus sampling on GPT-2 Large continuations; nucleus with came closest to human text on perplexity and repetition
- Order matters
- top-p after top-k works on the renormalised top-k distribution, so the combined set can be smaller than either rule alone (worked example below)
Worked example
Take a peaked distribution: Paris 0.790, located 0.048, a 0.036, the 0.029, home 0.018, then 79 tail tokens at 0.001 each (total 1).
| Rank | Token | p | Cumulative | In V₀.₉? | q = p / 0.903 |
|---|---|---|---|---|---|
| 1 | Paris | 0.790 | 0.790 | yes | 0.875 |
| 2 | located | 0.048 | 0.838 | yes | 0.053 |
| 3 | a | 0.036 | 0.874 | yes | 0.040 |
| 4 | the | 0.029 | 0.903 | yes (reaches 0.9) | 0.032 |
| 5 | home | 0.018 | 0.921 | no | 0 |
| 6–84 | tail × 79 | 0.001 each | 1.000 | no | 0 |
The same distribution under other settings
- 1Top-k = 2: keep Paris and located, mass 0.838; and .
- 2Top-k = 50: keep 5 named tokens and 45 tail tokens, mass 0.966. The tail still holds of the sampling mass.
- 3Top-k = 50, then top-p = 0.9: on the renormalised top-50 the running sums are 0.818, 0.868, 0.905, so only 3 tokens survive, one fewer than top-p alone.
1import numpy as np23def truncate(probs, top_k=0, top_p=1.0):4 """Hugging Face order: top-k first, then top-p on the renormalised survivors."""5 p = np.asarray(probs, dtype=np.float64)6 order = np.argsort(-p, kind="stable")7 keep = np.zeros(p.size, dtype=bool)8 n = p.size if top_k <= 0 else min(top_k, p.size)9 keep[order[:n]] = True10 if top_p < 1.0:11 q = p[order[:n]] / p[order[:n]].sum()12 m = int(np.searchsorted(np.cumsum(q), top_p) + 1) # smallest prefix with mass >= top_p13 keep[:] = False14 keep[order[:m]] = True15 out = np.where(keep, p, 0.0)16 return out / out.sum()1718# Paris, located, a, the, home, then a tail of 79 tokens at 0.001 each (sums to 1)19p = np.r_[0.790, 0.048, 0.036, 0.029, 0.018, np.full(79, 0.001)]20print(truncate(p, top_p=0.9)[:6].round(3)) # [0.875 0.053 0.04 0.032 0. 0. ]21print(truncate(p, top_k=2)[:6].round(3)) # [0.943 0.057 0. 0. 0. 0. ]22print((truncate(p, top_k=50) > 0).sum(), # 5023 (truncate(p, top_k=50, top_p=0.9) > 0).sum()) # 3Use when
- +Top-k: one integer, trivial to implement, hard cap on candidates; cheap on GPU with a partial sort.
- +Top-p: set size follows the model's confidence, so one value works across flat and peaked steps.
- +Both remove the tail exactly, which temperature alone cannot do.
Avoid when
- −Top-k: cuts good tokens on flat steps and admits junk on peaked ones.
- −Top-p: needs a full sort and cumulative sum per step; at high p on flat steps the nucleus can swell into the tail.
- −Both interact with temperature and with each other, so tuning one changes the effect of the others.
Check yourself
The top token has probability 0.93. Which tokens does top-p = 0.9 keep?
Check yourself
Why can top-k = 50 followed by top-p = 0.9 keep fewer tokens than top-p = 0.9 alone?
Remember
- ✓Top-k keeps a fixed count; top-p keeps the smallest prefix with mass ≥ p; both renormalise and sample.
- ✓A fixed k is too tight on flat steps and too loose on peaked ones; the nucleus adapts its size.
- ✓Top-p can still swell into the tail at high p or high T; Min-p scales the cut-off with the top token instead.
- ✓Hugging Face: temperature → top-k → top-p, defaults top_k = 50 and top_p = 1.0 with do_sample=True.
- ✓top_k = 1 or a tiny top_p is greedy; top_p = 1 and top_k = 0 switch truncation off.