Definition
Emergent ability "An ability is emergent if it is not present in smaller models but is present in larger models" (Wei et al., "Emergent Abilities of Large Language Models", TMLR 2022). Such an ability cannot be predicted by extrapolating the performance of smaller models: the score sits near chance across several sizes, then rises sharply. The word comes from Anderson's essay "More Is Different" (1972), where quantitative change produces qualitative change.
The question matters to anyone who picks, trains or regulates models. If abilities appear suddenly, you cannot test a small model and forecast what a large one will do. A capability that is harmless at 10B parameters may appear at 100B with no warning. Ganguli et al. ("Predictability and Surprise in Large Generative Models", 2022) put it as two facts that coexist: overall loss is predictable, specific capabilities are not.
Wei et al. collected examples from GPT-3, LaMDA, Gopher, Chinchilla and PaLM. Few-shot tasks such as multi-digit arithmetic, transliteration into the International Phonetic Alphabet, word unscrambling, MMLU and TruthfulQA stayed near chance until a model-size threshold, then improved. They also listed prompting methods that help only above a certain scale. Chain-of-thought prompting is the best known: Wei et al.'s chain-of-thought paper (2022) found it helped only at around 100B parameters; smaller models wrote fluent but illogical chains and gained nothing or lost accuracy.
BIG-bench (Srivastava et al., 2022; 204 tasks) called this "breakthrough" behaviour. It noted that breakthrough tasks often involve several steps or components, or use brittle metrics. The next section shows why those two features produce jumps.
Where a jump can come from
Suppose the probability of each correct answer token improves smoothly with scale. Score a task by exact match on an answer of tokens, and treat the tokens as independent for simplicity:
- per-token cross-entropy on the answer; a power law in model size
- probability that a given answer token is correct
- answer length in tokens (or number of reasoning steps that must all be right)
- exact match: 1 only if every token is right; a nonlinear metric
The edit distance is linear in and changes as smoothly as does. Exact match raises to the power . It stays near zero while is moderate and climbs fast once approaches 1. Multiple-choice grading behaves the same way: it scores 1 only when the right option's probability overtakes all the others, a threshold on a smooth quantity.
This is the argument of Schaeffer, Miranda and Koyejo, "Are Emergent Abilities of Large Language Models a Mirage?" (NeurIPS 2023). On GPT-3/InstructGPT arithmetic, the jumps seen under exact match became smooth, predictable curves under token edit distance. In their survey of BIG-bench, the large majority of emergent abilities (over 92% in their count) appeared under just two metrics, multiple-choice grade and exact string match. They also produced "emergence" in vision models on purpose, just by choosing a sharp metric.
Small test sets hide the early rise
- the model's true per-item success rate on the task
- number of test items
A model with true exact-match rate (the model above) scores exactly 0 on a 200-item test with probability . With 2,000 items that falls to 0.0007. Many BIG-bench tasks have only a few hundred items, so "absent in smaller models" can mean "below the test's resolution".
The main sources
- Definition
- Wei, Tay, Bommasani, Raffel, Zoph et al., "Emergent Abilities of Large Language Models", TMLR 2022; examples plotted against training FLOPs and parameter count
- Scale-gated prompting
- Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", NeurIPS 2022: gains appear only at around 100B parameters
- Benchmark
- BIG-bench (Srivastava et al., 2022): 204 tasks; describes "breakthrough" tasks, often multi-step or with brittle metrics
- Metric critique
- Schaeffer, Miranda & Koyejo, NeurIPS 2023: nonlinear or discontinuous metrics plus small test sets can produce apparent emergence
- Policy view
- Ganguli et al., FAccT 2022: smooth loss scaling alongside abrupt, hard-to-predict specific capabilities
- The reverse
- McKenzie et al., "Inverse Scaling: When Bigger Isn't Better" (2023): tasks where larger models do worse
The lesson for practice is in the two forecasts. A sharp metric throws away the information that small models are improving. A continuous metric keeps it, and a scaling-law fit on it predicts the large model well. With mixed item difficulty the exact-match curve is less sharp and the loss-based forecast is only approximate (71.8% vs 83.0% at the defaults), because averaging over items breaks the simple relation.
Worked example: one curve, three metrics
| N | ℓ | p = e^−ℓ | EM, L = 5 | EM, L = 10 | edit distance, L = 5 |
|---|---|---|---|---|---|
| 10⁸ | 2.000 | 0.135 | < 0.001 | < 0.001 | 4.32 |
| 10⁹ | 0.632 | 0.531 | 0.042 | 0.002 | 2.34 |
| 10¹⁰ | 0.200 | 0.819 | 0.368 | 0.135 | 0.91 |
| 10¹¹ | 0.063 | 0.939 | 0.729 | 0.531 | 0.31 |
Each decade of divides the loss by . The edit distance falls by a factor of 1.8–2.9 per decade throughout. Exact match for goes from 0.002 to 0.135 to 0.531: about a 76-fold rise across one decade, then 3.9-fold across the next. The same compounding applies to multi-step tasks. A model that gets each of 20 steps right with probability 0.95 completes the whole task of the time; at 0.99 per step it completes it 81.8% of the time.
1import numpy as np23def answer_scores(gold_logprobs):4 """Teacher-forced log p(gold token) at each answer position -> three metrics."""5 lp = np.asarray(gold_logprobs, dtype=float)6 return {7 "exact_match_prob": float(np.exp(lp.sum())), # p(all tokens right) = prod p_t8 "mean_token_prob": float(np.exp(lp).mean()), # smooth, per-token9 "answer_logprob": float(lp.sum()), # continuous, never stuck at 010 }1112# three small models scored on the same 5-token answer (per-token loss 2.0, 0.632, 0.2)13sizes = np.array([1e8, 1e9, 1e10])14losses = np.array([2.0, 0.632, 0.2])15for n, l in zip(sizes, losses):16 print(f"{n:.0e}", {k: round(v, 4) for k, v in answer_scores([-l] * 5).items()})17# 1e+08 {'exact_match_prob': 0.0, 'mean_token_prob': 0.1353, 'answer_logprob': -10.0}18# 1e+09 {'exact_match_prob': 0.0424, 'mean_token_prob': 0.5315, 'answer_logprob': -3.16}19# 1e+10 {'exact_match_prob': 0.3679, 'mean_token_prob': 0.8187, 'answer_logprob': -1.0}2021# forecast a 1e11 model: fit the per-token loss on log-log axes, then convert22b, a = np.polyfit(np.log10(sizes), np.log(losses), 1)23l_hat = np.exp(a + b * 11)24print(round(b / np.log(10), 3), round(l_hat, 4), round(np.exp(-5 * l_hat), 3))25# -0.5 0.0632 0.729 (exponent, predicted loss, predicted exact match)Where it's used
Model selection
Compare candidate models on per-token log-likelihood or partial credit as well as pass/fail, to see how far a smaller, cheaper model is from the threshold (see Model families and model selection).
Evaluation design
Use enough items that is well below the scores you care about, and report continuous metrics next to exact match (see LLM evaluation and benchmarking).
Training forecasts
Fit scaling curves on small runs using loss or log-likelihood of reference answers, then convert to the task metric, as in the figure.
Risk assessment
Test dangerous capabilities on every new model with strong elicitation (good prompts, chain of thought, tools), since they may appear between sizes.
Check yourself
Per-token accuracy on a 10-token answer improves from 0.5 to 0.9 across model sizes. What happens to exact match?
Check yourself
Why does a straight line through small models' exact-match scores forecast the large model badly in the figure?
Remember
- ✓Wei et al. (2022): an ability is emergent if it is absent in smaller models and present in larger ones, so it cannot be extrapolated.
- ✓Examples: multi-digit arithmetic, IPA transliteration, MMLU and gains from chain-of-thought prompting at about 100B parameters.
- ✓Schaeffer et al. (2023): all-or-nothing metrics (exact match, multiple-choice grade) turn smooth gains into jumps; EM = pᴸ.
- ✓Small test sets add a floor: a 200-item test cannot show scores below 0.5%.
- ✓Forecast with continuous metrics (log-likelihood, edit distance), decide with the task metric, and test each new model directly.