Definition
Speculative decoding is a way to sample from a large autoregressive model faster without changing its output distribution. A small draft model proposes tokens one at a time. The target then scores all of them in one forward pass. Each draft token is accepted with probability . At the first rejection, a replacement is sampled from the normalised residual and the rest is discarded. If all drafts pass, one bonus token comes from the target's last distribution. The method was published independently by Leviathan, Kalman and Matias, "Fast Inference from Transformers via Speculative Decoding" (Google, ICML 2023, arXiv November 2022), and by Chen, Borgeaud, Irving, Lespiau, Sifre and Jumper, "Accelerating Large Language Model Decoding with Speculative Sampling" (DeepMind, February 2023).
A Decoder-only transformer generates one token per forward pass, and each pass depends on the previous one. At small batch sizes that pass is limited by memory bandwidth, not arithmetic. Every step streams all the weights and the KV cache from HBM to compute a single new position. Running the same pass over 5 positions instead of 1 costs almost the same wall-clock time, because the weights are read once either way. The GPU has spare arithmetic and nothing to spend it on.
Speculative decoding spends that spare arithmetic on guesses. Many next tokens are easy: the end of a common word, a closing bracket, a phrase copied from the prompt. A model 10 to 100 times smaller predicts these as well as the big one. If the draft is right, the target confirms several tokens in the time it normally takes to produce one. If it is wrong, the target's own pass already contains the information needed to correct it, so at least one token still comes out of every call.
The idea has a precursor. Stern, Shazeer and Uszkoreit, "Blockwise Parallel Decoding for Deep Autoregressive Models" (NeurIPS 2018), added heads that predict several future tokens and kept the longest prefix that agreed with greedy decoding. That works only for greedy (argmax) decoding. The 2023 papers extended the idea to sampling with a rejection rule that leaves the target distribution exactly intact. That is what made it practical for chat models sampled at temperature above zero.
The acceptance rule
Fix one position. The target's next-token distribution there is and the draft's is . The draft samples . Then:
- target and draft distributions at the same position, given the same prefix (temperature, top-p etc. applied to both)
- the draft under-proposes : always accept
- the draft over-proposes : keep it only a fraction of the time
- the residual distribution: the probability mass the target wants and the draft did not supply
This is rejection sampling with a twist. Ordinary rejection sampling throws away rejected draws and tries again, which would need another target call. Here the rejected mass is replaced in one shot by sampling from exactly the part of that the accepted draws are missing.
Why the output is exactly p
Write . The probability that a draft is accepted is . So a rejection happens with probability . The residual's normaliser is . Now add up the two ways token can come out:
- acceptance probability; equals , one minus the total-variation distance
- for any :
The argument is per position and holds for any , however bad. A poor draft makes generation slower, never wrong. Positions after the first rejection are discarded because they were conditioned on a token that did not survive. Every kept token was conditioned on a prefix that is itself an exact sample from . By induction the whole sequence is distributed exactly as if had generated it alone. In greedy decoding, and are one-hot, and the rule reduces to "accept while the draft equals the target's argmax".
How many tokens per target call
Leviathan et al. analyse the simple case where each draft is accepted independently with the same probability . The number of accepted drafts before the first rejection is a geometric variable capped at . One more token always comes out: either the residual sample or the bonus.
- draft length: tokens proposed per step
- expected acceptance rate , measured on real prompts
- cost of one draft step relative to one target step; one speculative step costs target-step equivalents
- tokens per call saturate at ; at that is 5
The trade-off in is visible in the formula. A longer draft adds a term to the numerator, which shrinks geometrically. It also adds a fixed to the denominator. So there is an optimal that rises with and falls with . The price is extra arithmetic: the target scores positions per call, and the rejected ones are wasted work. The method pays off when spare compute is free, which is when decoding is memory-bound.
From the papers
- Leviathan et al. 2023
- T5-XXL (11B) target with smaller T5 drafts on translation and summarisation; reported a 2X–3X speedup over the standard T5X implementation, with identical outputs
- Chen et al. 2023
- Chinchilla 70B target and a 4B-parameter draft model; 2–2.5× decoding speedup in a distributed setup, with sample quality unchanged and no change to the target model
- Acceptance test
- , accept iff ; one uniform per draft token
- Target pass
- one forward over drafts returns distributions; the KV cache of rejected positions is rolled back in both models
- Draft choices
- a smaller model of the same family sharing the tokenizer (required: and must be over the same vocabulary); n-grams copied from the prompt (prompt lookup); extra heads on the target (Medusa); a light feature-level head (EAGLE)
- Hugging Face
- "assisted generation":
model.generate(..., assistant_model=draft);prompt_lookup_num_tokens=kuses n-grams from the prompt as drafts
Worked example: one position, four tokens
Take a 4-token vocabulary with target and draft over tokens A, B, C, D.
| token | p | q | min(p, q) | accept prob. min(1, p/q) | max(0, p − q) | residual p′ |
|---|---|---|---|---|---|---|
| A | 0.50 | 0.30 | 0.30 | 1 | 0.20 | 0.8 |
| B | 0.30 | 0.50 | 0.30 | 0.6 | 0 | 0 |
| C | 0.15 | 0.10 | 0.10 | 1 | 0.05 | 0.2 |
| D | 0.05 | 0.10 | 0.05 | 0.5 | 0 | 0 |
| sum | 1 | 1 | α = 0.75 | 0.25 = 1 − α | 1 |
Checking the output and the speedup
- 1The draft over-proposes B and D, so those are the only tokens that can be rejected. If the draft says B, it survives with probability .
- 2A rejection happens with probability in total: from B plus from D. The replacement is A with probability and C with . B and D are never resampled.
- 3Output probabilities: A , B , C , D . This is exactly .
- 4With and : tokens per target call.
- 5With a draft costing : speedup . Trying other lengths gives (), (), (, the optimum) and ().
1import numpy as np23rng = np.random.default_rng(0)4V = 50 # toy vocabulary5logits_p = rng.normal(0, 2.0, (V, V)) # target: a bigram model p(x_t | x_{t-1})6logits_q = logits_p + rng.normal(0, 1.0, (V, V)) # draft: a noisier copy of it78def softmax(z):9 z = z - z.max(-1, keepdims=True)10 e = np.exp(z)11 return e / e.sum(-1, keepdims=True)1213P, Q = softmax(logits_p), softmax(logits_q) # row i = next-token distribution after token i1415def target_forward(ctx): # ONE call scores every position of ctx in parallel16 return P[np.array(ctx)] # shape (len(ctx), V): p(. | ctx[:j+1]) for each j1718def speculative_step(ctx, gamma, rng):19 """Draft gamma tokens with q, verify with one target call. Returns the new tokens."""20 drafts, qs, last = [], [], ctx[-1]21 for _ in range(gamma): # serial, but the draft is cheap22 q = Q[last]23 last = rng.choice(V, p=q)24 drafts.append(last); qs.append(q)25 ps = target_forward(ctx + drafts)[len(ctx) - 1:] # p for positions 1..gamma+126 out = []27 for i, x in enumerate(drafts):28 p, q = ps[i], qs[i]29 if rng.random() < min(1.0, p[x] / q[x]): # accept with prob min(1, p/q)30 out.append(x)31 continue32 resid = np.maximum(p - q, 0.0) # reject: resample from norm(max(0, p - q))33 out.append(rng.choice(V, p=resid / resid.sum()))34 return out35 out.append(rng.choice(V, p=ps[gamma])) # all accepted: one bonus token from p36 return out3738gamma, calls, emitted = 4, 0, 039first = np.zeros(V) # distribution of the first token after context [7]40for _ in range(20000):41 new = speculative_step([7], gamma, rng)42 calls += 1; emitted += len(new); first[new[0]] += 14344alpha = np.minimum(P[7], Q[7]).sum() # acceptance rate of the first draft after token 745alpha_avg = np.minimum(P, Q).sum(-1).mean()46print(f"alpha after token 7: {alpha:.3f}, mean alpha over contexts: {alpha_avg:.3f}") # 0.530, 0.65247print(f"tokens per target call: {emitted / calls:.3f}") # 2.16348print(f"TV(first token, p(.|7)) = {0.5 * np.abs(first / first.sum() - P[7]).sum():.4f}") # 0.010249# For scale: 20000 direct samples from p(.|7) have TV 0.0148 to p, so the gap is pure noise.In the toy run, differs by context, from 0.530 after token 7 to 0.652 on average, so the i.i.d. formula is only a guide. Real deployments measure on their own traffic. Code, structured output and summarisation of the prompt accept many more drafts than open-ended creative text.
Beyond a separate draft model
A separate draft has costs. It must share the tokenizer, it uses memory, it keeps its own KV cache, and its predictions are made without the target's hidden state. Several later methods draft from the target itself.
Medusa (Cai et al., ICML 2024) adds a few extra decoding heads on the target's last hidden state. Head predicts the token positions ahead. Their top candidates form a tree, and tree attention verifies many candidate continuations in one pass with a custom attention mask. Medusa-1 trains only the heads on a frozen backbone and reports more than 2.2× speedup. Medusa-2 fine-tunes the backbone jointly and reports 2.3–3.6×. Medusa also proposes a "typical acceptance" rule that accepts plausible tokens without the exact rejection test. That is faster but no longer exactly .
EAGLE (Li, Wei, Zhang and Zhang, ICML 2024) drafts at the feature level. A single light transformer layer predicts the target's next second-to-top-layer feature from the current feature and the already-sampled next token. The target's own LM head then turns it into a token. Feeding in the sampled token removes the uncertainty about which branch the sequence took. It keeps the exact acceptance rule and verifies a tree of drafts. EAGLE-2 makes the tree shape depend on the draft's confidence.
Lookahead decoding (Fu, Bailis, Stoica and Zhang, ICML 2024) needs no draft at all. It runs Jacobi-style fixed-point iterations on a window of future positions, collects the n-grams these iterations produce in a pool, and verifies matching n-grams in the same forward pass. Prompt lookup is the simplest version: copy the n-gram that followed the current suffix earlier in the prompt. This works very well for editing and retrieval-augmented answers.
Use when
- +Exactly the target's output distribution, for sampling as well as greedy decoding
- +No change to the target model's weights or training
- +Typical 2–3× lower latency for single-stream decoding when is high
- +Works with any draft source: a small model, prompt n-grams, extra heads or a feature-level head
Avoid when
- −Extra compute: target positions per call, with rejected positions wasted
- −Gains shrink at large batch sizes where decoding is compute-bound
- −A separate draft needs the same tokenizer, extra memory and its own KV cache
- −Speed depends on , which varies by task and prompt; creative sampling at high temperature accepts less
Where it's used
Interactive chat latency
Lower time per output token for single users, where decoding is memory-bound.
Code completion and editing
Highly predictable continuations and copied spans give high acceptance; prompt lookup is especially effective.
Serving engines
vLLM, TensorRT-LLM and Hugging Face TGI support draft models, n-gram drafting and EAGLE- or Medusa-style heads.
Check yourself
A draft token x has q(x) = 0.4 while the target gives p(x) = 0.1. What happens?
Check yourself
With α = 0.8 and a free draft (c = 0), how many tokens per target call can you expect at most, however long the draft?
Remember
- ✓A cheap draft proposes tokens. The target scores them all in one memory-bound forward pass, which costs about the same as scoring one token.
- ✓Accept each draft with prob. . At the first rejection, resample from and stop. If all pass, add a bonus token from .
- ✓: the output is exactly the target's distribution for any draft.
- ✓With acceptance rate : tokens per call , speedup that .
- ✓Medusa, EAGLE, lookahead decoding and prompt lookup replace the separate draft model. Gains shrink at large batch sizes.