Definition
LLM evaluation is the measurement of what a language model can do, under a fixed and fully specified protocol: a dataset, a prompt format, a scoring rule and a statistic with its uncertainty. There are four main protocols. Intrinsic evaluation reports perplexity or bits per byte on held-out text. Likelihood-based multiple choice ranks fixed answer options by their log-probability, as in MMLU (Hendrycks et al., ICLR 2021). Generative evaluation samples an answer and checks it, by exact match on GSM8K (Cobbe et al., 2021) or by unit tests on HumanEval with the pass@k estimator of Chen et al. ("Evaluating Large Language Models Trained on Code", 2021). Preference evaluation compares two models' answers using an LLM judge (MT-Bench) or crowd votes (Chatbot Arena; Zheng et al., NeurIPS 2023; Chiang et al., ICML 2024), fitted with a Bradley–Terry model.
A Language modeling loss tells you how well a model predicts text. It does not tell you whether the model can answer a chemistry question, write a working function or follow an instruction. Before GPT-3 that gap was closed by fine-tuning: GLUE and SuperGLUE scored a pre-trained encoder such as BERT after training it on each task. Brown et al. ("Language Models are Few-Shot Learners", 2020) changed the setup. They evaluated one frozen model on dozens of tasks by putting zero, one or a few solved examples in the prompt. Since then a benchmark score has meant "this checkpoint, this prompt, this scorer", with no task-specific training at all.
That shift created the problems this article is about. Scores depend on prompt wording and answer extraction. Test sets leak into web-scale training data. Benchmarks saturate within a year or two, so new ones keep appearing: MMLU (2020), HumanEval and GSM8K (2021), BIG-bench (2022), GPQA and SWE-bench (2023). Open-ended chat quality has no answer key at all, so it is judged by humans or by other models. Benchmarks are often small, a few hundred items, so a two-point gap can be pure noise.
Intrinsic evaluation: perplexity and bits per byte
- number of tokens in the held-out text
- number of UTF-8 bytes in the same text
- effective branching factor: a model with PPL 10 is as unsure as a uniform choice among 10 tokens
- bits per byte; independent of the tokenizer, used by The Pile (Gao et al. 2020)
Perplexity is cheap, smooth and has no prompt to tune, so it is the standard signal during Pretraining and continued pretraining. It has two limits. It is per token, so two models with different tokenizers are not comparable: a tokenizer with longer tokens spreads the same information over fewer, harder tokens. Bits per byte normalises by the text instead. For example, 1,000 tokens at mean NLL 2.3 nats give PPL ; if the text is 4,200 bytes, that is bits per byte. The second limit is that perplexity predicts downstream ability only loosely. Within one model family, lower loss usually means better benchmarks. Across families, data mixtures and post-training it often does not, and post-training such as RLHF often raises perplexity on generic web text while making the model far more useful.
Multiple choice by log-likelihood
For a question with fixed options, the cleanest protocol never lets the model generate. The harness builds the context (instructions, few-shot examples, the question), appends each option in turn, and reads off the log-probability of the option's tokens. The prediction is the option with the highest score. No parsing is needed and the result is deterministic. EleutherAI's lm-evaluation-harness and the early Open LLM Leaderboard used this protocol for ARC, HellaSwag and MMLU.
- context: instructions, few-shot examples and the question
- answer option , tokenized as a continuation of
- total log-probability of the option; a sum, so longer options are penalised
- lm-evaluation-harness divides by the option's character count; GPT-3 divided by token count
The sum is biased toward short options: every extra token adds another negative log-probability. On HellaSwag, where the four endings are full sentences of different lengths, this matters. Say the wrong ending has 2 tokens with total and the right one has 6 tokens with total . The raw sum picks the wrong one, while per-token averages ( vs ) pick the right one. Brown et al. used per-token normalisation for most tasks. For ARC, OpenBookQA and RACE they divided by the option's unconditional probability given only "Answer:", , which removes the prior that some answers are simply common strings.
MMLU is usually scored differently. The options are shown in the prompt labelled A–D, and the harness compares the probabilities of the single letter tokens " A", " B", " C", " D" after "Answer:". This tests whether the model can map its knowledge onto a letter, which small base models do poorly. The Hugging Face team showed in 2023 that three popular MMLU implementations (the original code, HELM and the Eleuther harness) gave noticeably different scores, and even different rankings, for the same models. A score without its protocol cannot be compared.
Generative evaluation and pass@k
Math and code need the model to produce the answer. GSM8K, 8.5K grade-school word problems needing 2 to 8 steps, is scored by extracting the final number and checking exact match. Chain-of-thought prompting (Wei et al., 2022) uses 8 worked exemplars. MATH (Hendrycks et al., NeurIPS 2021) asks for a final answer in , which must be normalised before comparison. For code, matching against a reference is useless because many programs are correct. HumanEval runs the completion against hidden unit tests.
Sampling makes one more question natural: if the model may try times, how often does at least one attempt pass? Chen et al. define pass@k as the expected fraction of problems solved by at least one of samples. Drawing exactly samples per problem gives a high-variance estimate. Instead they draw samples (they used ), count the that pass, and compute:
- samples drawn per problem; must be at least k
- samples that pass every unit test
- probability that a random -subset of the samples contains no correct one
- the numerically stable form from the paper; binomials overflow for n = 200
Why is it unbiased? The samples are i.i.d. with success probability . A uniformly random -subset of them is itself i.i.d. samples, so the probability that the subset has no success, averaged over the draw, is exactly . The obvious plug-in alternative with is biased low. The function is concave, so by Jensen's inequality . The bias grows with and shrinks with . At both estimators reduce to .
pass@k for one problem with n = 10 samples, c = 3 correct
- 1: , the same as .
- 2: and , so pass@5 . The product form agrees: .
- 3Plug-in estimate: , which is 8.5 points lower.
- 4: any 8 of the 10 samples must include a correct one, so pass@8 exactly. The plug-in gives .
Temperature matters. Chen et al. found the best temperature rose with : about for pass@1 and for pass@100, since diversity helps when any one success counts. Codex-12B solved 28.8% of HumanEval with one sample. Many current reports give greedy pass@1 instead, which is a different number. SWE-bench (Jimenez et al., ICLR 2024) extends the idea to real repositories: the model must produce a patch that makes the issue's failing tests pass without breaking the others.
Benchmarks most often reported
- MMLU
- Hendrycks et al., ICLR 2021. 57 subjects from elementary maths to law, 4 options, 14,042 test questions, usually 5-shot. GPT-3 175B scored 43.9% few-shot against 25% chance
- HellaSwag
- Zellers et al., ACL 2019. Pick the right continuation of a scene among 4; wrong endings chosen by adversarial filtering against BERT-era models. Humans above 95%; 10,042 validation items. Scored by length-normalised log-likelihood
- ARC
- Clark et al., 2018. 7,787 grade-school science questions; the Challenge set (2,590) keeps those that both a retrieval and a word co-occurrence baseline got wrong
- GSM8K
- Cobbe et al., 2021. 8.5K grade-school word problems (7,473 train, 1,319 test), 2 to 8 steps, integer answers, exact match
- MATH
- Hendrycks et al., NeurIPS 2021. 12,500 competition problems (7,500 train, 5,000 test), 7 subjects, difficulty levels 1 to 5, final answer in . MATH-500 is the 500-problem test subset of Lightman et al. (2023)
- HumanEval
- Chen et al., 2021. 164 hand-written Python functions with docstrings, on average 7.7 unit tests each; pass@k with n = 200 samples
- TruthfulQA
- Lin, Hilton and Evans, ACL 2022. 817 questions in 38 categories built to trigger common misconceptions. Best model then truthful on 58% vs 94% for humans; larger models were often less truthful
- BIG-bench
- Srivastava et al., 2022. 204 tasks from 450 authors at 132 institutions. BIG-Bench Hard (Suzgun et al., 2022) keeps 23 tasks on which earlier models fell below the average human rater
- GPQA
- Rein et al., 2023. 448 graduate-level multiple-choice questions in biology, physics and chemistry. Experts 65%, skilled non-experts 34% even with unrestricted web access for 30+ minutes. Diamond subset: 198 questions
- SWE-bench
- Jimenez et al., ICLR 2024. 2,294 GitHub issues from 12 Python repositories; graded by the repository's own tests. SWE-bench Verified (OpenAI, 2024) is a human-validated subset of 500
- MT-Bench
- Zheng et al., NeurIPS 2023. 80 two-turn questions in 8 categories (writing, roleplay, extraction, reasoning, math, coding, STEM, humanities), graded 1 to 10 by GPT-4
LLM-as-a-judge and Chatbot Arena
Open-ended answers have no key. Zheng et al. ("Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023) asked GPT-4 to grade answers, either one at a time on a 1–10 scale or as a pairwise comparison. GPT-4's verdicts agreed with human experts over 80% of the time, about as often as two humans agreed with each other. The same paper catalogued the judge's biases. Position bias: the judge prefers whichever answer comes first (or second), so verdicts can flip when the order is swapped. Verbosity bias: longer answers win even when the extra length adds nothing. Self-enhancement: a judge may favour answers written by its own model family. Judges are also weak at grading maths and reasoning they cannot solve themselves. The standard fixes are to run both orders and call it a tie when they disagree, to give the judge a reference answer, and to control for length. Length-controlled AlpacaEval (Dubois et al., 2024) regresses out the length difference before computing win rates.
Chatbot Arena (Chiang et al., ICML 2024) replaces the judge with crowds. A user types a prompt, two anonymous models answer, and the user votes for one, a tie, or both bad. The model names are revealed only after the vote. The pairwise outcomes are fitted with the Bradley–Terry model, the same model that underlies the reward model in RLHF:
- latent strength of model i, fitted by maximum likelihood (logistic regression on battle outcomes)
- the same strength on the Elo scale; 400/ln 10 ≈ 173.7 points per unit of log-odds
- means i wins 64.0% of battles; 200 means 76.0%; 400 means 90.9%
The leaderboard first used online Elo updates, . For example, with a 1000-rated model that beats a 1100-rated one had expected score and gains points. Online Elo depends on the order of battles and assumes players' strengths drift, which static model weights do not. Bradley–Terry maximum likelihood uses all battles at once and does not depend on their order. The Arena reports bootstrap confidence intervals for each rating. Models whose intervals overlap are effectively tied. The Arena later added "style control", which adds answer length and markdown use as covariates so that ratings are not driven by formatting.
Few-shot, zero-shot and prompt sensitivity
Base models are usually evaluated few-shot: solved examples in the prompt show the format and the answer space. MMLU is typically 5-shot and GSM8K 8-shot with chain-of-thought. Instruction-tuned and chat models are often evaluated zero-shot through their chat template, because few-shot examples formatted as plain text can confuse them. Neither is "the" score. Reports should state the shot count, the template and whether chain-of-thought was used.
Scores are sensitive to choices that should not matter. Zhao et al. ("Calibrate Before Use", ICML 2021) found that the prompt format, the choice of few-shot examples and even their order could move GPT-3 accuracy from near chance to near state of the art. They traced this to majority-label, recency and common-token biases. Lu et al. (ACL 2022) found the same for example order. Sclar et al. (2023) varied only formatting (separators, casing, spacing) and saw accuracy spreads of up to 76 points for LLaMA-2-13B on some tasks. Multiple-choice answers also depend on which letter holds the right option. Robust evaluations average over several prompts or permute the options, and report the spread as well as the mean.
Contamination
A benchmark measures generalisation only if its test items were not in training. Web-scale corpora make that hard to guarantee: benchmarks are posted on GitHub, discussed on forums and copied into other datasets. Brown et al. checked GPT-3 with 13-gram overlap between benchmarks and training data, and reported results on "clean" subsets. Overlap checks miss paraphrases and translations. Other signals include unusually high likelihood of the exact test text (Min-K% Prob, Shi et al., 2023), accuracy that drops on freshly written look-alike problems (GSM1k, Zhang et al., 2024), and accuracy that falls for problems published after the training cutoff (LiveCodeBench, Jain et al., 2024). BIG-bench embeds a canary string in its files so that training pipelines can filter them out. Fine-tuning on benchmark-like data is a related, softer problem: the model gets better at the format, not at the skill.
How big is a benchmark difference?
An accuracy is a mean of Bernoulli outcomes. Its standard error is . At 70% on MMLU's 14,042 questions that is 0.39 points. At 50% on GPQA Diamond's 198 questions it is 3.55 points, so a 95% interval is about points. HumanEval's 164 problems give about points of standard error at 50%. Miller ("Adding Error Bars to Evals", 2024) argues that every eval should report these, and should use clustered standard errors when questions come in groups, such as several questions about one passage.
- whether each model got question i right
- discordant questions: only A right, only B right. Questions both get right or both get wrong carry no information about the difference
- smaller than the unpaired whenever the two models find the same questions hard
Two models evaluated on the same questions are paired. Hard questions are hard for both, so most of the per-question variance cancels in the difference. Treating the two accuracies as independent overstates the uncertainty, and the correct test (McNemar's test, or a paired bootstrap that resamples questions) is often much sharper. The same logic gives the smallest difference worth trying to detect. With 80% power at that is about . If that exceeds the gap you care about, you need more questions, not more decimal places.
Worked example: is model B really better?
Two models answer the same 500 multiple-choice questions. A gets 70.0% and B gets 74.0%. Is the 4-point gap real? The per-question outcomes form a 2×2 table:
| A vs B | B right | B wrong | total |
|---|---|---|---|
| A right | 320 | 30 | 350 (70.0%) |
| A wrong | 50 | 100 | 150 |
| total | 370 (74.0%) | 130 | 500 |
Three ways to put an error bar on Δ = +4.0 points
- 1Unpaired (wrong here). pp. The 95% CI is pp, with and : "not significant".
- 2Paired. Only the discordant questions vary. , so pp. The CI is pp, with and .
- 3McNemar's test uses the discordant counts alone: with 1 degree of freedom, so . The exact binomial version (is 30 out of 80 consistent with ?) gives .
- 4Paired bootstrap with 10,000 resamples of the 500 questions (the code below): 95% CI pp, in line with the CLT interval.
- 5Conclusion: B is better at the 5% level, but the interval is wide. The true gap could plausibly be anywhere from about half a point to about seven and a half points. A claim of "4.0 points better" should come with that interval.
1import numpy as np2import torch3from transformers import AutoModelForCausalLM, AutoTokenizer45# ---- 1. multiple choice by log-likelihood (the acc / acc_norm protocol) ----6tok = AutoTokenizer.from_pretrained("gpt2")7model = AutoModelForCausalLM.from_pretrained("gpt2").eval()89@torch.no_grad()10def option_logprob(context: str, option: str) -> float:11 """Sum of log p(option tokens | context). Context and option are tokenized12 separately and concatenated, so the option's tokens are exactly its own."""13 ctx = tok(context).input_ids14 opt = tok(option, add_special_tokens=False).input_ids15 ids = torch.tensor([ctx + opt])16 logp = model(ids).logits[0, :-1].log_softmax(-1) # row t predicts token t+117 targets = ids[0, 1:]18 lp = logp.gather(-1, targets[:, None])[:, 0]19 return lp[-len(opt):].sum().item() # keep only the option's tokens2021def score_mc(context: str, options: list[str], gold: int) -> tuple[bool, bool]:22 s = np.array([option_logprob(context, " " + o) for o in options])23 chars = np.array([len(o) for o in options], dtype=float)24 return int(s.argmax()) == gold, int((s / chars).argmax()) == gold # acc, acc_norm2526ctx = "A man is sitting on a roof. He"27opts = ["is using wrap to wrap a pair of skis.", "is ripping level tiles off.",28 "is holding a rubik's cube.", "starts pulling up roofing on a roof."]29print(score_mc(ctx, opts, gold=3))3031# ---- 2. pass@k: the unbiased estimator of Chen et al. (2021) ----32def pass_at_k(n: int, c: int, k: int) -> float:33 """1 - C(n-c, k) / C(n, k), computed as a stable product."""34 if n - c < k:35 return 1.036 return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))3738print(f"{pass_at_k(10, 3, 1):.4f} {pass_at_k(10, 3, 5):.4f} {1 - 0.7 ** 5:.4f}") # 0.3000 0.9167 0.8319 (plug-in)3940# ---- 3. paired bootstrap CI for acc(B) - acc(A) on the same questions ----41def paired_bootstrap(a: np.ndarray, b: np.ndarray, reps: int = 10_000, seed: int = 0):42 d = b.astype(float) - a.astype(float) # per-question difference in {-1, 0, 1}43 se = d.std(ddof=1) / np.sqrt(len(d)) # paired CLT standard error44 rng = np.random.default_rng(seed)45 idx = rng.integers(0, len(d), size=(reps, len(d))) # resample questions, not answers46 boot = d[idx].mean(axis=1)47 return d.mean(), se, np.percentile(boot, [2.5, 97.5])4849# the worked example: 320 both right, 30 only A, 50 only B, 100 both wrong50a = np.array([1] * 320 + [1] * 30 + [0] * 50 + [0] * 100)51b = np.array([1] * 320 + [0] * 30 + [1] * 50 + [0] * 100)52delta, se, ci = paired_bootstrap(a, b)53print(f"delta={delta:+.3f} paired SE={se:.4f} bootstrap 95% CI={ci.round(3)}")54# delta=+0.040 paired SE=0.0178 bootstrap 95% CI=[0.006 0.076]5556# ---- 4. Bradley-Terry ratings from pairwise battles (MM algorithm, Hunter 2004) ----57def bradley_terry(wins: np.ndarray, iters: int = 1000) -> np.ndarray:58 """wins[i, j] = battles i won against j (count a tie as half a win for each side)."""59 games = wins + wins.T60 p = np.ones(len(wins))61 for _ in range(iters):62 p = wins.sum(axis=1) / (games / (p[:, None] + p[None, :])).sum(axis=1)63 p /= np.exp(np.log(p).mean()) # fix the scale: geometric mean 164 return 1000 + 400 * np.log10(p) # the Elo-like scale the Arena reports6566wins = np.array([[0, 60, 75], # model 0 beat model 1 60 times, model 2 75 times67 [40, 0, 55],68 [25, 45, 0]], dtype=float)69print(bradley_terry(wins).round(1)) # [1085.3 987.8 926.9]Use when
- +Log-likelihood multiple choice is deterministic, cheap and needs no answer parsing; it gives signal even for small base models
- +Generative exact match and unit tests check what users care about, the final answer or a working program
- +Unbiased pass@k from n ≥ k samples has far lower variance than drawing exactly k samples
- +LLM judges and Arena votes cover open-ended quality that has no answer key
- +Bradley–Terry ratings put many models on one scale from sparse pairwise comparisons
Avoid when
- −Scores depend on prompt, shot count and scoring details; harnesses disagree on the same model
- −Static test sets leak into web-scale training data and saturate within a few years
- −Small benchmarks (164 to a few hundred items) have standard errors of 2 to 4 points
- −LLM judges show position, verbosity and self-enhancement biases
- −Arena ratings reflect the prompts and tastes of the people who vote, and reward style as well as substance
- −Perplexity is not comparable across tokenizers and tracks downstream ability only loosely
Where it's used
Pretraining ablations
Validation loss and bits per byte, plus log-likelihood multiple choice (HellaSwag, ARC), give smooth signals on small models where generative scores are still near zero.
Post-training regression tests
A fixed suite of generative tasks (GSM8K, HumanEval, instruction following), run on every checkpoint with paired comparisons against the previous one.
Model selection
A private set drawn from your own traffic, graded by tests, rubrics or an LLM judge with both answer orders, is what predicts production quality.
Public leaderboards
Chatbot Arena and benchmark tables compare released models; read them with their confidence intervals and protocols.
Check yourself
A problem gets n = 10 samples and exactly c = 1 passes. What is the unbiased pass@5 estimate?
Check yourself
On GSM8K's 1,319 test problems, model A scores 90.0% and model B scores 91.5% (B gets 20 more right). When is B's lead statistically significant at the 5% level?
Remember
- ✓Perplexity is per token, so it compares only models with one tokenizer; bits per byte fixes that, but neither predicts downstream ability reliably across model families.
- ✓Multiple choice by log-likelihood picks ; length normalisation (
acc_norm) removes the bias toward short options. - ✓pass@k from samples is unbiased; the plug-in is biased low by Jensen's inequality.
- ✓LLM judges agree with humans about as often as humans agree with each other on MT-Bench, but they have position, verbosity and self-enhancement biases. Swap the answer order and control for length.
- ✓Chatbot Arena fits Bradley–Terry, , to crowd votes and reports bootstrap intervals; 100 rating points means a 64% win rate.
- ✓Contamination, prompt format, shot count and answer extraction can each move scores by more than the gaps being reported.
- ✓Report and a paired confidence interval: the SE of an accuracy is (3.55 points on 198 questions at 50%), and paired tests on discordant questions are much sharper than unpaired ones.