Top-k and top-p sampling

Top-k keeps the k most likely tokens; top-p (nucleus, Holtzman et al. 2020) keeps the smallest set whose probability reaches p. Both zero the tail and renormalise, but only top-p adapts the set size to the model's confidence.

7 min read610 wordscore0 viewsupdated 2026-10-21

Definition

Top-k and top-p sampling are truncation rules applied to the next-token distribution before sampling. Top-k keeps only the kk highest-probability tokens; it was popularised for story generation by Fan, Lewis and Dauphin ("Hierarchical Neural Story Generation", ACL 2018) and used with k=40k=40 for the GPT-2 samples (Radford et al., 2019). Top-p, or nucleus sampling, keeps the smallest set of top tokens whose cumulative probability is at least pp; it was introduced by Holtzman, Buys, Du, Forbes and Choi in "The Curious Case of Neural Text Degeneration" (ICLR 2020). Both renormalise the survivors and sample from them.

Pure sampling from a language model has a tail problem. Each of the tens of thousands of unlikely tokens has a tiny probability, but together they hold several percent of the mass. Over a few hundred steps, at least one junk token is almost certain, and the text that follows conditions on it.

Temperature sampling cannot fix this: lowering TT shrinks the tail but also removes diversity among good tokens. Truncation removes the tail outright and leaves the head untouched. It runs after temperature and before the random draw; Min-p sampling is a third rule with the same role.

The two rules

Vk={the k tokens with largest pi},Vp=arg⁡min⁡V′  ∣V′∣  s.t. ∑i∈V′pi≥pV_k=\{\text{the }k\text{ tokens with largest }p_i\},\qquad V_p=\arg\min_{V'}\;|V'|\ \ \text{s.t.}\ \sum_{i\in V'}p_i\ge p
pip_i
next-token probability after temperature
VkV_k
top-k set: fixed size, variable mass
VpV_p
nucleus: fixed mass, variable size; built by adding tokens in descending order until the running sum reaches p
qi=pi 1[i∈V]∑j∈Vpjq_i=\frac{p_i\,\mathbb 1[i\in V]}{\sum_{j\in V}p_j}
VV
the kept set, VkV_k, VpV_p, or both applied in sequence
qiq_i
renormalised probability actually sampled from; every cut token gets exactly 0

Renormalisation scales every survivor by the same factor 1/∑j∈Vpj1/\sum_{j\in V}p_j, so ratios among kept tokens are unchanged. In logit space, implementations set cut logits to −∞-\infty and let softmax do the rest.

truncation runs after temperature and before sampling (Hugging Face order)logits ÷ Tsoftmaxtop-ktop-prenormalisesample"My favourite food is" (flat)pcum1pizza0.1290.1292sushi0.1170.2463pasta0.1060.3524chocolate0.0870.4395tacos0.0780.5176curry0.0710.5887ramen0.0640.6528steak0.0580.7109cheese0.0530.76310salad0.0430.80611soup0.0390.84512bread0.0350.88013rice0.0320.91214Tuesday0.0040.916k = 5 keeps 0.517 of the mass, cuts 8 foodsp = 0.9 keeps 13 tokens (0.912), all foods"The capital of France is" (peaked)pcum1Paris0.7900.7902located0.0480.8383a0.0360.8744the0.0290.9035home0.0180.9206one0.0130.9347not0.0090.9428Lyon0.0050.9489in0.0040.95210called0.0030.95511actually0.0020.95712also0.0020.95913banana<.0010.96014purple<.0010.960k = 5 keeps 5 tokens, including "home"p = 0.9 keeps only 4 tokens (0.903)dashed: top-k cut, k = 5 · shaded: nucleus, p = 0.9 · rose: junk or wrong · tail not shown
fig.The same two rules on two contexts. Flat ("My favourite food is"): top token 0.129; k = 5 keeps only 0.517 of the mass and cuts 8 plausible foods, while the p = 0.9 nucleus holds 13 tokens (0.912), all foods. Peaked ("The capital of France is"): Paris has 0.790; k = 5 still keeps five tokens, while the nucleus stops at four (cumulative 0.903).

Why a fixed k fails

Holtzman et al. argued that the right number of candidates changes from step to step. After "My favourite food is", dozens of continuations are fine, and a small kk cuts good ones. After "The capital of France is", one token dominates, and a large kk admits dozens of poor ones. Top-p adapts: the nucleus is small when the model is confident and large when it is not.

Top-p has its own weak spot. When the distribution is flat, reaching a high pp can require dipping into the tail, and the set size jumps. In the figure below, raising pp from 0.90 to 0.95 on the flat context grows the nucleus from 13 tokens to 384. That sensitivity, worse at high temperature, motivated Min-p sampling.

Settings worth knowing (documented behaviour as of 2024–2025)

Hugging Face transformers
top_k=50 and top_p=1.0 by default, active only with do_sample=True; the warpers run temperature, then top-k, then top-p, and always keep at least one token
OpenAI Chat Completions
top_p (default 1), no top_k; the docs recommend changing temperature or top_p, not both
Anthropic Messages API
exposes both top_k and top_p; the docs suggest most users only need temperature
Holtzman et al. (2020)
compared greedy, beam, pure, top-k and nucleus sampling on GPT-2 Large continuations; nucleus with p=0.95p=0.95 came closest to human text on perplexity and repetition
Order matters
top-p after top-k works on the renormalised top-k distribution, so the combined set can be smaller than either rule alone (worked example below)
"My favourite food is …" · p = 0.90 · T = 1outline = p, fill = renormalised qp → qpizza0.129 → 0.142sushi0.117 → 0.128pasta0.106 → 0.116chocolate0.087 → 0.095tacos0.078 → 0.086curry0.071 → 0.078ramen0.064 → 0.070steak0.058 → 0.064cheese0.053 → 0.058salad0.043 → 0.047soup0.039 → 0.043bread0.035 → 0.039rice0.032 → 0.035Tuesday0.004 → cutpurple0.003 → cutquickly0.002 → cuttail0.079 → cut1,000 junk tokens, p = 7.9e-5 each; 0 keptcumulative mass vs rank (log scale)00.511101001000p · k · tail starts · cuttokens kept13 of 1016original mass kept0.912plausible tokens kept13 / 13junk mass after renorm0.0000P(≥1 junk in 50 steps)0.000
context
truncation
fig.Two toy contexts, each with 16 named tokens plus 1,000 junk tail tokens. Default (flat context, top-p = 0.9): 13 of 1,016 tokens kept, 0.912 of the mass, all 13 foods kept, junk mass 0, P(junk in 50 steps) = 0. With no truncation the junk mass is 0.0882 and P(junk in 50 steps) = 0.990. Raise p to 0.95: 384 tokens kept, junk 0.0402, P = 0.872. Switch to the peaked context: p = 0.9 keeps 4 tokens; top-k = 50 keeps 34 tail tokens, junk 0.0082, P = 0.338; top-k 50 → top-p 0.9 keeps just 3.

Worked example

Take a peaked distribution: Paris 0.790, located 0.048, a 0.036, the 0.029, home 0.018, then 79 tail tokens at 0.001 each (total 1).

RankTokenpCumulativeIn V₀.₉?q = p / 0.903
1Paris0.7900.790yes0.875
2located0.0480.838yes0.053
3a0.0360.874yes0.040
4the0.0290.903yes (reaches 0.9)0.032
5home0.0180.921no0
6–84tail × 790.001 each1.000no0
Nucleus with p = 0.9: add tokens until the running sum reaches 0.9, then divide by the kept mass 0.903

The same distribution under other settings

  1. 1Top-k = 2: keep Paris and located, mass 0.838; q=0.790/0.838=0.943q=0.790/0.838=0.943 and 0.0570.057.
  2. 2Top-k = 50: keep 5 named tokens and 45 tail tokens, mass 0.966. The tail still holds 0.045/0.966=4.7%0.045/0.966=4.7\% of the sampling mass.
  3. 3Top-k = 50, then top-p = 0.9: on the renormalised top-50 the running sums are 0.818, 0.868, 0.905, so only 3 tokens survive, one fewer than top-p alone.
python
1import numpy as np
2
3def truncate(probs, top_k=0, top_p=1.0):
4 """Hugging Face order: top-k first, then top-p on the renormalised survivors."""
5 p = np.asarray(probs, dtype=np.float64)
6 order = np.argsort(-p, kind="stable")
7 keep = np.zeros(p.size, dtype=bool)
8 n = p.size if top_k <= 0 else min(top_k, p.size)
9 keep[order[:n]] = True
10 if top_p < 1.0:
11 q = p[order[:n]] / p[order[:n]].sum()
12 m = int(np.searchsorted(np.cumsum(q), top_p) + 1) # smallest prefix with mass >= top_p
13 keep[:] = False
14 keep[order[:m]] = True
15 out = np.where(keep, p, 0.0)
16 return out / out.sum()
17
18# Paris, located, a, the, home, then a tail of 79 tokens at 0.001 each (sums to 1)
19p = np.r_[0.790, 0.048, 0.036, 0.029, 0.018, np.full(79, 0.001)]
20print(truncate(p, top_p=0.9)[:6].round(3)) # [0.875 0.053 0.04 0.032 0. 0. ]
21print(truncate(p, top_k=2)[:6].round(3)) # [0.943 0.057 0. 0. 0. 0. ]
22print((truncate(p, top_k=50) > 0).sum(), # 50
23 (truncate(p, top_k=50, top_p=0.9) > 0).sum()) # 3
Top-k then top-p in the Hugging Face order (numpy; runs as is). In transformers the same thing is generate(..., do_sample=True, top_k=50, top_p=0.9)

Use when

  • +Top-k: one integer, trivial to implement, hard cap on candidates; cheap on GPU with a partial sort.
  • +Top-p: set size follows the model's confidence, so one value works across flat and peaked steps.
  • +Both remove the tail exactly, which temperature alone cannot do.

Avoid when

  • −Top-k: cuts good tokens on flat steps and admits junk on peaked ones.
  • −Top-p: needs a full sort and cumulative sum per step; at high p on flat steps the nucleus can swell into the tail.
  • −Both interact with temperature and with each other, so tuning one changes the effect of the others.

Check yourself

The top token has probability 0.93. Which tokens does top-p = 0.9 keep?

Check yourself

Why can top-k = 50 followed by top-p = 0.9 keep fewer tokens than top-p = 0.9 alone?

Remember

  • ✓Top-k keeps a fixed count; top-p keeps the smallest prefix with mass ≥ p; both renormalise and sample.
  • ✓A fixed k is too tight on flat steps and too loose on peaked ones; the nucleus adapts its size.
  • ✓Top-p can still swell into the tail at high p or high T; Min-p scales the cut-off with the top token instead.
  • ✓Hugging Face: temperature → top-k → top-p, defaults top_k = 50 and top_p = 1.0 with do_sample=True.
  • ✓top_k = 1 or a tiny top_p is greedy; top_p = 1 and top_k = 0 switch truncation off.
← back to Generative AI roadmap