# Perplexity

> Perplexity is exp of the mean next-token negative log-likelihood on held-out text. It only compares models with the same tokenizer, text and context protocol; bits per byte fixes the tokenizer, and a sliding window fixes truncated context.

Source: https://www.mlfoundry.in/learn/perplexity  
Section: Generative AI › Evaluation of Generative Models  
Level: core · 8 min read · updated 2026-10-21

**Perplexity:** is the exponential of a language model's average negative log-likelihood per token on a text: $\mathrm{PPL}=\exp\!\big(-\tfrac1N\sum_i\ln p_\theta(x_i\mid x_{<i})\big)$. Jelinek, Mercer, Bahl and Baker introduced it in 1977 to measure the difficulty of speech-recognition tasks. It reads as an effective branching factor: a model with perplexity 20 is as uncertain, on average, as a uniform choice among 20 tokens.

**Language modeling** derives the quantity from the training loss. This article is about using it as an evaluation number: when it is the right metric, what it depends on besides the model, and how to compute it so two numbers can be compared.

In practice perplexity shows up in four places. It is the headline metric of pretraining runs and scaling studies. It is a regression test after **Quantization**, **Pruning** or a fine-tune: a quantized checkpoint whose WikiText-2 perplexity rises by 0.1 has lost little, one that rises by 2 is broken. It is used as a data filter, scoring web text with a small reference model. And it is an input to detectors of memorised or machine-written text.

It is a poor metric for chat quality. After instruction tuning and RLHF, models deliberately move probability away from typical web text, so their perplexity on it rises while their usefulness goes up. For assistant behaviour use task metrics, **LLM-as-a-judge** or **Human evaluation and arena-style ranking**.

## Definition and units

$$
\mathrm{PPL}=\exp\!\Big(\underbrace{-\tfrac{1}{N}\textstyle\sum_{i=1}^{N}\ln p_\theta(x_i\mid x_{<i})}_{\mathcal L\ \text{(nats/token)}}\Big)=2^{\,\mathcal L/\ln 2}
$$

- $N$: number of **scored** tokens; prompt or padding tokens excluded from the loss are not counted
- $p_\theta(x_i\mid x_{<i})$: probability the model gave the token that actually came next, from one forward pass with teacher forcing
- $\mathcal L$: mean cross-entropy, the same number the training loop reports; $\mathcal L/\ln2$ is bits per token

The bounds are clean. A model that is always certain and right scores 1. A uniform model over $V$ tokens scores exactly $V$: 50,257 for GPT-2's vocabulary. Any token with probability 0 makes it infinite, which is why evaluated models must smooth. Because perplexity is an exponential, small loss changes look large: going from 2.0 to 1.9 nats is a 9.5% drop in perplexity (7.39 to 6.69).

> Figure: Top: one forward pass over 'The cat sat on the mat.' yields the probability of each actual next token. The surprisals −ln p sum to 12.822 nats over 7 tokens, a mean of 1.832, so perplexity is e^1.832 = 6.24. The same total over the 23 UTF-8 bytes is 0.804 bits per byte. Had a tokenizer split the text into 5 or 10 tokens with the same total, per-token perplexity would read 12.99 or 3.60. Bottom: a 16-token text with window 8. Non-overlapping chunks score the first token of each chunk with no context. A sliding window with stride 4 scores only the 4 new tokens of each window, so every token after the first window sees at least 4 tokens of context, at the price of 3 passes and 24 processed tokens instead of 2 and 16.

## What else the number depends on

**Tokenizer.** Perplexity is per token, and tokenizers cut the same text into different numbers of tokens. A larger vocabulary makes fewer, harder predictions, which raises per-token perplexity even when the model compresses the text better. Compare across tokenizers with a byte-normalised measure:

$$
\mathrm{BPB}=\frac{N_\text{tok}}{N_\text{bytes}}\cdot\frac{\mathcal L}{\ln 2},\qquad \mathrm{PPL}_\text{token}=2^{\,\mathrm{BPB}\,\cdot\,N_\text{bytes}/N_\text{tok}}
$$

- $N_\text{bytes}$: UTF-8 bytes of the evaluated text (independent of the tokenizer)
- $\mathrm{BPB}$: bits per byte; the Pile benchmark (Gao et al., 2020) reports it for this reason. Bits per character is the same idea per character

**Text and preprocessing.** WikiText-2 versus C4, raw versus detokenised, with or without document boundaries: each changes the number. So does whether the first token of a document is scored, and whether a BOS token is prepended.

**Context protocol.** Models have a maximum window $W$. Long test sets are cut into pieces, and how they are cut changes how much context each prediction gets:

$$
c_i^{\text{chunks}}=i\bmod W,\qquad c_i^{\text{sliding}}\ \ge\ W-S\ \ (i\ge W),\qquad \text{cost}\approx N\cdot\frac{W}{S}\ \text{tokens}
$$

- $c_i$: number of preceding tokens the model sees when scoring token $i$
- $S$: stride: each window advances $S$ tokens and scores only its last $S$; the first $W-S$ are context with labels masked

Non-overlapping chunks are cheap but score every chunk's first tokens with almost no context, which inflates perplexity. A sliding window with $S<W$ gives every scored token at least $W-S$ tokens of context and approaches the full-context value as $S\to1$, at about $W/S$ times the compute.

**Reference points**

- **Hugging Face perplexity guide:** GPT-2 large on WikiText-2 with a 1,024-token window: 19.44 with non-overlapping chunks (the GPT-2 paper reports 19.93), 16.45 with stride 512. Same model, same text
- **Quantization papers:** Llama-2-7B in FP16 is commonly reported at about 5.47 on WikiText-2 with 2,048-token segments; 4-bit methods are judged by how far they move this number
- **Data filtering:** CCNet (Wenzek et al., 2020) scored Common Crawl with a 5-gram KenLM model trained on Wikipedia and split each language into head, middle and tail thirds by perplexity
- **Membership tests:** Min-K% Prob (Shi et al., 2024) flags likely training documents by the mean log-probability of their k% least likely tokens; very low perplexity on a benchmark is a contamination warning sign

> Figure: A toy LM scores a 1,024-token document; a token's loss falls as 2.2 + 3/√(c + 1) nats with c tokens of context, times a fixed per-token difficulty. Defaults (W = 64, S = 32): non-overlapping chunks give perplexity 18.96 with mean context 31.5; the sliding window gives 14.96 (mean NLL 2.705 nats) with 31 passes and 1.94× the compute. Stride 8 brings it to 14.23 at 7.56×. With W = 256, chunks give 13.64 and stride 16 gives 11.91 at 12.25×; unbounded context would give 11.31. Same model, same tokens: only the protocol changed.

## Worked example: the same text, three tokenizers

Take the diagram's sentence, 23 bytes, with a total cost of 12.822 nats. Suppose three models assign the same total probability to the text but use different tokenizers:

Same total NLL (12.822 nats = 18.50 bits), different token counts

| Tokenizer | Tokens | Bytes/token | Mean NLL (nats) | Per-token PPL | Bits per byte |
| --- | --- | --- | --- | --- | --- |
| Coarse (large vocab) | 5 | 4.60 | 2.564 | 12.99 | 0.804 |
| Medium | 7 | 3.29 | 1.832 | 6.24 | 0.804 |
| Fine (small vocab) | 10 | 2.30 | 1.282 | 3.60 | 0.804 |

Per-token perplexity spans a factor of 3.6 for models that are equally good at predicting this text. Bits per byte is identical. Report BPB, or nats per byte, whenever tokenizers differ, and quote per-token perplexity only between models that share a tokenizer, such as one checkpoint before and after quantization.

```python
import math, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "gpt2-large"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name).eval()

def perplexity(text, window=1024, stride=512):
    ids = tok(text, return_tensors="pt").input_ids
    n = ids.size(1)
    nll_sum, n_scored, prev_end = 0.0, 0, 0
    for begin in range(0, n, stride):
        end = min(begin + window, n)
        trg = end - prev_end                    # new tokens scored in this window
        x = ids[:, begin:end]
        labels = x.clone()
        labels[:, :-trg] = -100                 # context only: excluded from the loss
        with torch.no_grad():
            loss = model(x, labels=labels).loss # mean NLL over scored, shifted labels
        k = (labels[:, 1:] != -100).sum().item()
        nll_sum += loss.item() * k
        n_scored += k
        prev_end = end
        if end == n:
            break
    mean_nll = nll_sum / n_scored
    n_bytes = len(text.encode("utf-8"))
    return math.exp(mean_nll), nll_sum / math.log(2) / n_bytes   # PPL, bits per byte
```

Sliding-window perplexity with Hugging Face transformers (the pattern from its perplexity guide). Not run here: torch and transformers were not available in the scratch environment

> **Numbers that look comparable but are not:** A perplexity is only meaningful with its protocol: test set and version, tokenizer, window, stride, BOS handling and which tokens are scored. Papers that differ in any of these routinely disagree by several points for the same checkpoint. For instruction-tuned models, also state whether the chat template was applied and whether prompt tokens were masked; for conditional perplexity of an answer, score only the answer tokens.

> **Lower is not always better:** Perplexity rewards matching the reference distribution. A model that has memorised the test set scores near 1; a model trained to refuse or to write in a different register scores worse on web text. Treat a sudden drop on a public benchmark as possible contamination, and treat perplexity on generated text (the model scoring its own outputs) as a measure of blandness, not quality: greedy, repetitive text has very low perplexity.

- **Compression regression tests:** same tokenizer and protocol before and after **Quantization**, **Pruning** or **Knowledge distillation**; small deltas are meaningful here
- **Pretraining and scaling:** validation loss or BPB on a held-out mix tracks progress and fits scaling laws
- **Data curation:** score candidate documents with a reference LM and drop the tail, as in CCNet; see **Dataset curation for fine-tuning**
- **Domain fit:** a model's perplexity on your own corpus is a quick check of how foreign the domain is before choosing between **Fine-tuning vs RAG vs prompting**

**Question:** On the same test text, model A (3.6 bytes per token) has perplexity 8.0 and model B (4.4 bytes per token) has perplexity 10.0. Which predicts the text better?

- A. A, because its perplexity is lower
- B. B: it needs 0.755 bits per byte versus A's 0.833
- C. They are equal
- D. It cannot be computed without the vocabulary sizes

**Answer:** B. B: it needs 0.755 bits per byte versus A's 0.833 — Convert to bits per byte: A is $\log_2 8/3.6=3/3.6=0.833$, B is $\log_2 10/4.4=3.322/4.4=0.755$. B makes fewer, harder predictions, so its per-token perplexity is higher, but it encodes the text in fewer total bits.

**Question:** You evaluate one checkpoint with window 1,024 and change the stride from 1,024 to 256. What happens?

- A. Perplexity rises because more tokens are scored
- B. Nothing; perplexity is a property of the model
- C. Perplexity falls because every scored token now sees at least 768 tokens of context, and compute rises about 4×
- D. Perplexity falls because fewer tokens are scored

**Answer:** C. Perplexity falls because every scored token now sees at least 768 tokens of context, and compute rises about 4× — The same $N$ tokens are scored in both cases. Only their context changes: with stride $S$ each scored token sees at least $W-S$ previous tokens. Cost is about $W/S=4$ times the tokens processed.

**Key points**

- $\mathrm{PPL}=\exp(\text{mean NLL per scored token})$; 1 is perfect, a uniform model over $V$ tokens scores $V$.
- It is comparable only with the same tokenizer, text, preprocessing and context protocol; report the protocol with the number.
- Across tokenizers use bits per byte: $\mathcal L/\ln2\cdot N_\text{tok}/N_\text{bytes}$.
- Sliding-window evaluation (stride $S<W$) gives each token at least $W-S$ context and lowers perplexity at about $W/S$ times the compute.
- Good for pretraining, compression regressions, data filtering and contamination checks; poor for judging aligned chat models.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
