Perplexity

Perplexity is exp of the mean next-token negative log-likelihood on held-out text. It only compares models with the same tokenizer, text and context protocol; bits per byte fixes the tokenizer, and a sliding window fixes truncated context.

8 min read743 wordscore0 viewsupdated 2026-10-21

Definition

Perplexity is the exponential of a language model's average negative log-likelihood per token on a text: PPL=exp⁡ ⁣(−1N∑iln⁡pθ(xi∣x<i))\mathrm{PPL}=\exp\!\big(-\tfrac1N\sum_i\ln p_\theta(x_i\mid x_{<i})\big). Jelinek, Mercer, Bahl and Baker introduced it in 1977 to measure the difficulty of speech-recognition tasks. It reads as an effective branching factor: a model with perplexity 20 is as uncertain, on average, as a uniform choice among 20 tokens.

Language modeling derives the quantity from the training loss. This article is about using it as an evaluation number: when it is the right metric, what it depends on besides the model, and how to compute it so two numbers can be compared.

In practice perplexity shows up in four places. It is the headline metric of pretraining runs and scaling studies. It is a regression test after Quantization, Pruning or a fine-tune: a quantized checkpoint whose WikiText-2 perplexity rises by 0.1 has lost little, one that rises by 2 is broken. It is used as a data filter, scoring web text with a small reference model. And it is an input to detectors of memorised or machine-written text.

It is a poor metric for chat quality. After instruction tuning and RLHF, models deliberately move probability away from typical web text, so their perplexity on it rises while their usefulness goes up. For assistant behaviour use task metrics, LLM-as-a-judge or Human evaluation and arena-style ranking.

Definition and units

PPL=exp⁡ ⁣(−1N∑i=1Nln⁡pθ(xi∣x<i)⏟L (nats/token))=2 L/ln⁡2\mathrm{PPL}=\exp\!\Big(\underbrace{-\tfrac{1}{N}\textstyle\sum_{i=1}^{N}\ln p_\theta(x_i\mid x_{<i})}_{\mathcal L\ \text{(nats/token)}}\Big)=2^{\,\mathcal L/\ln 2}
NN
number of scored tokens; prompt or padding tokens excluded from the loss are not counted
pθ(xi∣x<i)p_\theta(x_i\mid x_{<i})
probability the model gave the token that actually came next, from one forward pass with teacher forcing
L\mathcal L
mean cross-entropy, the same number the training loop reports; L/ln⁡2\mathcal L/\ln2 is bits per token

The bounds are clean. A model that is always certain and right scores 1. A uniform model over VV tokens scores exactly VV: 50,257 for GPT-2's vocabulary. Any token with probability 0 makes it infinite, which is why evaluated models must smooth. Because perplexity is an exponential, small loss changes look large: going from 2.0 to 1.9 nats is a 9.5% drop in perplexity (7.39 to 6.69).

PPL = exp(−(1/N) Σ ln p(xᵢ | x<ᵢ)): score every token, average, exponentiate1 · one forward pass gives p(next token | prefix) at every position"The"p = 0.043.219"␣cat"p = 0.023.912"␣sat"p = 0.102.303"␣on"p = 0.450.799"␣the"p = 0.600.511"␣mat"p = 0.251.386"."p = 0.500.693bottom number = −ln p in nats (surprisal); stronger red = more surprising · ␣ = leading spacetotal NLLΣ −ln p = 12.822mean over 7 tokens12.822 / 7 = 1.832perplexitye^1.832 = 6.24bits per byte (23 B)12.822/ln2/23 = 0.804same 12.822 nats split into 5 tokens → PPL 12.99; into 10 tokens → 3.60; bits per byte stays 0.8042 · a 16-token text, window W = 8: chunks vs sliding window with stride S = 4 (numbers = context length)chunks, pass 101234567scores 0–7, ctx 0–7chunks, pass 201234567scores 8–15, ctx 0–7slide, pass 101234567scores 0–7slide, pass 24567scores 8–11, ctx 4–7slide, pass 34567scores 12–15, ctx 4–7grey = context only (not scored) · slide: 3 passes, 24 tokens processed; chunks: 2, 16
fig.Top: one forward pass over 'The cat sat on the mat.' yields the probability of each actual next token. The surprisals −ln p sum to 12.822 nats over 7 tokens, a mean of 1.832, so perplexity is e^1.832 = 6.24. The same total over the 23 UTF-8 bytes is 0.804 bits per byte. Had a tokenizer split the text into 5 or 10 tokens with the same total, per-token perplexity would read 12.99 or 3.60. Bottom: a 16-token text with window 8. Non-overlapping chunks score the first token of each chunk with no context. A sliding window with stride 4 scores only the 4 new tokens of each window, so every token after the first window sees at least 4 tokens of context, at the price of 3 passes and 24 processed tokens instead of 2 and 16.

What else the number depends on

Tokenizer. Perplexity is per token, and tokenizers cut the same text into different numbers of tokens. A larger vocabulary makes fewer, harder predictions, which raises per-token perplexity even when the model compresses the text better. Compare across tokenizers with a byte-normalised measure:

BPB=NtokNbytes⋅Lln⁡2,PPLtoken=2 BPB ⋅ Nbytes/Ntok\mathrm{BPB}=\frac{N_\text{tok}}{N_\text{bytes}}\cdot\frac{\mathcal L}{\ln 2},\qquad \mathrm{PPL}_\text{token}=2^{\,\mathrm{BPB}\,\cdot\,N_\text{bytes}/N_\text{tok}}
NbytesN_\text{bytes}
UTF-8 bytes of the evaluated text (independent of the tokenizer)
BPB\mathrm{BPB}
bits per byte; the Pile benchmark (Gao et al., 2020) reports it for this reason. Bits per character is the same idea per character

Text and preprocessing. WikiText-2 versus C4, raw versus detokenised, with or without document boundaries: each changes the number. So does whether the first token of a document is scored, and whether a BOS token is prepended.

Context protocol. Models have a maximum window WW. Long test sets are cut into pieces, and how they are cut changes how much context each prediction gets:

cichunks=i mod W,cisliding ≥ W−S  (i≥W),cost≈N⋅WS tokensc_i^{\text{chunks}}=i\bmod W,\qquad c_i^{\text{sliding}}\ \ge\ W-S\ \ (i\ge W),\qquad \text{cost}\approx N\cdot\frac{W}{S}\ \text{tokens}
cic_i
number of preceding tokens the model sees when scoring token ii
SS
stride: each window advances SS tokens and scores only its last SS; the first W−SW-S are context with labels masked

Non-overlapping chunks are cheap but score every chunk's first tokens with almost no context, which inflates perplexity. A sliding window with S<WS<W gives every scored token at least W−SW-S tokens of context and approaches the full-context value as S→1S\to1, at about W/SW/S times the compute.

Reference points

Hugging Face perplexity guide
GPT-2 large on WikiText-2 with a 1,024-token window: 19.44 with non-overlapping chunks (the GPT-2 paper reports 19.93), 16.45 with stride 512. Same model, same text
Quantization papers
Llama-2-7B in FP16 is commonly reported at about 5.47 on WikiText-2 with 2,048-token segments; 4-bit methods are judged by how far they move this number
Data filtering
CCNet (Wenzek et al., 2020) scored Common Crawl with a 5-gram KenLM model trained on Wikipedia and split each language into head, middle and tail thirds by perplexity
Membership tests
Min-K% Prob (Shi et al., 2024) flags likely training documents by the mean log-probability of their k% least likely tokens; very low perplexity on a benchmark is a contamination warning sign
PPL = exp(mean NLL) over N = 1024 tokens · window W = 64, stride S = 3203264064128192256context seen per scored token (positions 0–255): chunks vs sliding (≥ W − S = 32)token positionfull context 11.311016221×2×4×8×16×perplexity vs tokens run through the model (× N), S = W … W/16W = 64chunks (S = W)18.96sliding, S = 3214.96mean NLL (nats)2.705forward passes31compute cost1.94×mean context46.5toy LM: loss_i(c) = m_i·(2.2 + 3/√(c+1)) nats, m_i lognormal with mean 1, fixed seed
window W
fig.A toy LM scores a 1,024-token document; a token's loss falls as 2.2 + 3/√(c + 1) nats with c tokens of context, times a fixed per-token difficulty. Defaults (W = 64, S = 32): non-overlapping chunks give perplexity 18.96 with mean context 31.5; the sliding window gives 14.96 (mean NLL 2.705 nats) with 31 passes and 1.94× the compute. Stride 8 brings it to 14.23 at 7.56×. With W = 256, chunks give 13.64 and stride 16 gives 11.91 at 12.25×; unbounded context would give 11.31. Same model, same tokens: only the protocol changed.

Worked example: the same text, three tokenizers

Take the diagram's sentence, 23 bytes, with a total cost of 12.822 nats. Suppose three models assign the same total probability to the text but use different tokenizers:

TokenizerTokensBytes/tokenMean NLL (nats)Per-token PPLBits per byte
Coarse (large vocab)54.602.56412.990.804
Medium73.291.8326.240.804
Fine (small vocab)102.301.2823.600.804
Same total NLL (12.822 nats = 18.50 bits), different token counts

Per-token perplexity spans a factor of 3.6 for models that are equally good at predicting this text. Bits per byte is identical. Report BPB, or nats per byte, whenever tokenizers differ, and quote per-token perplexity only between models that share a tokenizer, such as one checkpoint before and after quantization.

python
1import math, torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4name = "gpt2-large"
5tok = AutoTokenizer.from_pretrained(name)
6model = AutoModelForCausalLM.from_pretrained(name).eval()
7
8def perplexity(text, window=1024, stride=512):
9 ids = tok(text, return_tensors="pt").input_ids
10 n = ids.size(1)
11 nll_sum, n_scored, prev_end = 0.0, 0, 0
12 for begin in range(0, n, stride):
13 end = min(begin + window, n)
14 trg = end - prev_end # new tokens scored in this window
15 x = ids[:, begin:end]
16 labels = x.clone()
17 labels[:, :-trg] = -100 # context only: excluded from the loss
18 with torch.no_grad():
19 loss = model(x, labels=labels).loss # mean NLL over scored, shifted labels
20 k = (labels[:, 1:] != -100).sum().item()
21 nll_sum += loss.item() * k
22 n_scored += k
23 prev_end = end
24 if end == n:
25 break
26 mean_nll = nll_sum / n_scored
27 n_bytes = len(text.encode("utf-8"))
28 return math.exp(mean_nll), nll_sum / math.log(2) / n_bytes # PPL, bits per byte
Sliding-window perplexity with Hugging Face transformers (the pattern from its perplexity guide). Not run here: torch and transformers were not available in the scratch environment

Where it's used

Compression regression tests

same tokenizer and protocol before and after Quantization, Pruning or Knowledge distillation; small deltas are meaningful here

Pretraining and scaling

validation loss or BPB on a held-out mix tracks progress and fits scaling laws

Data curation

score candidate documents with a reference LM and drop the tail, as in CCNet; see Dataset curation for fine-tuning

Domain fit

a model's perplexity on your own corpus is a quick check of how foreign the domain is before choosing between Fine-tuning vs RAG vs prompting

Check yourself

On the same test text, model A (3.6 bytes per token) has perplexity 8.0 and model B (4.4 bytes per token) has perplexity 10.0. Which predicts the text better?

Check yourself

You evaluate one checkpoint with window 1,024 and change the stride from 1,024 to 256. What happens?

Remember

  • ✓PPL=exp⁡(mean NLL per scored token)\mathrm{PPL}=\exp(\text{mean NLL per scored token}); 1 is perfect, a uniform model over VV tokens scores VV.
  • ✓It is comparable only with the same tokenizer, text, preprocessing and context protocol; report the protocol with the number.
  • ✓Across tokenizers use bits per byte: L/ln⁡2⋅Ntok/Nbytes\mathcal L/\ln2\cdot N_\text{tok}/N_\text{bytes}.
  • ✓Sliding-window evaluation (stride S<WS<W) gives each token at least W−SW-S context and lowers perplexity at about W/SW/S times the compute.
  • ✓Good for pretraining, compression regressions, data filtering and contamination checks; poor for judging aligned chat models.
← back to Generative AI roadmap