Definition
Perplexity is the exponential of a language model's average negative log-likelihood per token on a text: . Jelinek, Mercer, Bahl and Baker introduced it in 1977 to measure the difficulty of speech-recognition tasks. It reads as an effective branching factor: a model with perplexity 20 is as uncertain, on average, as a uniform choice among 20 tokens.
Language modeling derives the quantity from the training loss. This article is about using it as an evaluation number: when it is the right metric, what it depends on besides the model, and how to compute it so two numbers can be compared.
In practice perplexity shows up in four places. It is the headline metric of pretraining runs and scaling studies. It is a regression test after Quantization, Pruning or a fine-tune: a quantized checkpoint whose WikiText-2 perplexity rises by 0.1 has lost little, one that rises by 2 is broken. It is used as a data filter, scoring web text with a small reference model. And it is an input to detectors of memorised or machine-written text.
It is a poor metric for chat quality. After instruction tuning and RLHF, models deliberately move probability away from typical web text, so their perplexity on it rises while their usefulness goes up. For assistant behaviour use task metrics, LLM-as-a-judge or Human evaluation and arena-style ranking.
Definition and units
- number of scored tokens; prompt or padding tokens excluded from the loss are not counted
- probability the model gave the token that actually came next, from one forward pass with teacher forcing
- mean cross-entropy, the same number the training loop reports; is bits per token
The bounds are clean. A model that is always certain and right scores 1. A uniform model over tokens scores exactly : 50,257 for GPT-2's vocabulary. Any token with probability 0 makes it infinite, which is why evaluated models must smooth. Because perplexity is an exponential, small loss changes look large: going from 2.0 to 1.9 nats is a 9.5% drop in perplexity (7.39 to 6.69).
What else the number depends on
Tokenizer. Perplexity is per token, and tokenizers cut the same text into different numbers of tokens. A larger vocabulary makes fewer, harder predictions, which raises per-token perplexity even when the model compresses the text better. Compare across tokenizers with a byte-normalised measure:
- UTF-8 bytes of the evaluated text (independent of the tokenizer)
- bits per byte; the Pile benchmark (Gao et al., 2020) reports it for this reason. Bits per character is the same idea per character
Text and preprocessing. WikiText-2 versus C4, raw versus detokenised, with or without document boundaries: each changes the number. So does whether the first token of a document is scored, and whether a BOS token is prepended.
Context protocol. Models have a maximum window . Long test sets are cut into pieces, and how they are cut changes how much context each prediction gets:
- number of preceding tokens the model sees when scoring token
- stride: each window advances tokens and scores only its last ; the first are context with labels masked
Non-overlapping chunks are cheap but score every chunk's first tokens with almost no context, which inflates perplexity. A sliding window with gives every scored token at least tokens of context and approaches the full-context value as , at about times the compute.
Reference points
- Hugging Face perplexity guide
- GPT-2 large on WikiText-2 with a 1,024-token window: 19.44 with non-overlapping chunks (the GPT-2 paper reports 19.93), 16.45 with stride 512. Same model, same text
- Quantization papers
- Llama-2-7B in FP16 is commonly reported at about 5.47 on WikiText-2 with 2,048-token segments; 4-bit methods are judged by how far they move this number
- Data filtering
- CCNet (Wenzek et al., 2020) scored Common Crawl with a 5-gram KenLM model trained on Wikipedia and split each language into head, middle and tail thirds by perplexity
- Membership tests
- Min-K% Prob (Shi et al., 2024) flags likely training documents by the mean log-probability of their k% least likely tokens; very low perplexity on a benchmark is a contamination warning sign
Worked example: the same text, three tokenizers
Take the diagram's sentence, 23 bytes, with a total cost of 12.822 nats. Suppose three models assign the same total probability to the text but use different tokenizers:
| Tokenizer | Tokens | Bytes/token | Mean NLL (nats) | Per-token PPL | Bits per byte |
|---|---|---|---|---|---|
| Coarse (large vocab) | 5 | 4.60 | 2.564 | 12.99 | 0.804 |
| Medium | 7 | 3.29 | 1.832 | 6.24 | 0.804 |
| Fine (small vocab) | 10 | 2.30 | 1.282 | 3.60 | 0.804 |
Per-token perplexity spans a factor of 3.6 for models that are equally good at predicting this text. Bits per byte is identical. Report BPB, or nats per byte, whenever tokenizers differ, and quote per-token perplexity only between models that share a tokenizer, such as one checkpoint before and after quantization.
1import math, torch2from transformers import AutoModelForCausalLM, AutoTokenizer34name = "gpt2-large"5tok = AutoTokenizer.from_pretrained(name)6model = AutoModelForCausalLM.from_pretrained(name).eval()78def perplexity(text, window=1024, stride=512):9 ids = tok(text, return_tensors="pt").input_ids10 n = ids.size(1)11 nll_sum, n_scored, prev_end = 0.0, 0, 012 for begin in range(0, n, stride):13 end = min(begin + window, n)14 trg = end - prev_end # new tokens scored in this window15 x = ids[:, begin:end]16 labels = x.clone()17 labels[:, :-trg] = -100 # context only: excluded from the loss18 with torch.no_grad():19 loss = model(x, labels=labels).loss # mean NLL over scored, shifted labels20 k = (labels[:, 1:] != -100).sum().item()21 nll_sum += loss.item() * k22 n_scored += k23 prev_end = end24 if end == n:25 break26 mean_nll = nll_sum / n_scored27 n_bytes = len(text.encode("utf-8"))28 return math.exp(mean_nll), nll_sum / math.log(2) / n_bytes # PPL, bits per byteWhere it's used
Compression regression tests
same tokenizer and protocol before and after Quantization, Pruning or Knowledge distillation; small deltas are meaningful here
Pretraining and scaling
validation loss or BPB on a held-out mix tracks progress and fits scaling laws
Data curation
score candidate documents with a reference LM and drop the tail, as in CCNet; see Dataset curation for fine-tuning
Domain fit
a model's perplexity on your own corpus is a quick check of how foreign the domain is before choosing between Fine-tuning vs RAG vs prompting
Check yourself
On the same test text, model A (3.6 bytes per token) has perplexity 8.0 and model B (4.4 bytes per token) has perplexity 10.0. Which predicts the text better?
Check yourself
You evaluate one checkpoint with window 1,024 and change the stride from 1,024 to 256. What happens?
Remember
- ✓; 1 is perfect, a uniform model over tokens scores .
- ✓It is comparable only with the same tokenizer, text, preprocessing and context protocol; report the protocol with the number.
- ✓Across tokenizers use bits per byte: .
- ✓Sliding-window evaluation (stride ) gives each token at least context and lowers perplexity at about times the compute.
- ✓Good for pretraining, compression regressions, data filtering and contamination checks; poor for judging aligned chat models.