Definition
Phi is Microsoft Research's series of small language models whose central bet is that data quality can substitute for scale. It began with phi-1 in Gunasekar et al., "Textbooks Are All You Need" (2023), a 1.3B-parameter Python model trained on "textbook-quality" data. That data was web code filtered by a classifier trained on GPT-4 quality labels, plus synthetic textbooks and exercises written by GPT-3.5. Later releases applied the recipe to general language: phi-1.5 (Li et al., 2023), phi-2 (2.7B, December 2023), the Phi-3 family (Abdin et al., 2024; phi-3-mini is 3.8B) and Phi-4 (14B, Abdin et al., December 2024). Architecturally they are ordinary decoder-only transformers. The differences are in the data and in how long they train.
Scaling laws had made model size the default lever. Kaplan et al. (2020) and then Hoffmann et al. (2022, "Chinchilla") described loss as a smooth function of parameters and tokens , measured on web-scale text of roughly fixed quality. Chinchilla's popular takeaway was that a compute-optimal model should see about 20 tokens per parameter. These laws treat a token as a token. The Phi papers asked how much the distribution of the tokens matters. If most web text is boilerplate, ads, or code without context, a model may spend most of its capacity on material that teaches little.
phi-1 was the test case. Its training corpus was about 7B tokens, and the model saw about 50B tokens in total, roughly 8 passes. On HumanEval it reached 50.6% pass@1, and 55.5% on MBPP. For comparison, StarCoder (15.5B parameters, 1T training tokens) scored 33.6%. A 350M-parameter sibling, phi-1-small, reached 45% on HumanEval. The result was surprising enough that much of the paper is spent on contamination checks.
The recipe: filter, then synthesise
Filtering. The phi-1 authors took the deduplicated Python subset of The Stack and StackOverflow, about 35B tokens. They asked GPT-4 to rate about 100K samples by "educational value for a student whose goal is to learn basic coding concepts". They then trained a random-forest classifier on embeddings from a pretrained code model to predict that rating, and used it to score the whole pool. The kept portion was about 6B tokens. GPT-4 is far too expensive to label 35B tokens, so the trick is to use it as a teacher for a cheap filter.
Synthesis. They added about 1B tokens of GPT-3.5-generated Python "textbooks": explanations interleaved with code. For fine-tuning there was a separate set of about 180M tokens of exercises, function signatures with docstrings to complete. The hard part of synthetic data is diversity. An LLM asked a thousand times for "a Python textbook" writes nearly the same one. The authors injected randomness by constraining topics and target audiences. phi-1.5 scaled this up with about 20B tokens of synthetic text on common sense and general knowledge, seeded from 20K hand-picked topics.
- the strong model's judgement of educational value, obtained only for a small sample
- a cheap classifier (random forest in phi-1) on frozen embeddings, run over the whole pool
- threshold chosen to keep a target fraction; lower τ keeps more tokens of lower average quality
- a pretrained model's embedding of the document; phi-1 used a code language model
From phi-1 to Phi-4
phi-1.5 kept phi-1's 1.3B architecture and moved from code to general reasoning. It trained on 150B tokens: phi-1's 7B corpus plus about 20B new synthetic tokens. The report claims natural-language performance comparable to models 5× larger. Its variants with a 95B-token filtered web set showed that adding filtered web data on top of the synthetic set helped. phi-2 (2.7B) was released through a blog post rather than a paper. It trained on 1.4T tokens, multiple passes over a 250B-token mixture of GPT-3.5 synthetic text and GPT-4-assessed web data from Falcon RefinedWeb and SlimPajama, on 96 A100s for 14 days. It was initialised by scaling up the knowledge of phi-1.5. It was a base model, without instruction tuning or RLHF.
Phi-3 (April 2024) went from proof of concept to product. phi-3-mini has 3.8B parameters, 32 layers, and the Llama 2 tokenizer. It trained on 3.3T tokens and scored 69% on MMLU and 8.38 on MT-Bench. Quantised to 4 bits it takes about 1.8 GB and runs on an iPhone 14 at more than 12 tokens per second. Training had two phases. Phase 1 was mostly filtered web text, for general knowledge. Phase 2 mixed even more heavily filtered web text with synthetic data aimed at reasoning. The report calls this a data-optimal regime. For a small model, it removes web content that adds facts but not reasoning. Its example is the result of a particular day's Premier League game. phi-3-small (7B) and phi-3-medium (14B) trained on 4.8T tokens and reached 75% and 78% on MMLU.
Phi-4 (14B, December 2024) made synthetic data the core of pretraining. About 40% of the roughly 10T training tokens came from 50 types of synthetic datasets, about 400B unweighted tokens in total. They were built by multi-agent prompting, self-revision and instruction reversal, and seeded from curated web, books and code. On STEM question-answering benchmarks such as GPQA and MATH, it outperformed GPT-4o, its teacher for much of that data. To address contamination concerns, the authors also tested it on the November 2024 AMC-10/12 maths contests, which postdate its training data. Post-training used SFT and Direct preference optimization (DPO), including DPO pairs built by "pivotal token search", which finds the single tokens that most change the probability of a correct answer.
Configurations reported by the authors
- phi-1
- 1.3B; 24 layers, , 32 heads of 64, MLP 8192, rotary dimension 32, context 2048, FlashAttention, parallel attention/MLP; about 4 days on 8 A100s; HumanEval 50.6%, MBPP 55.5%
- phi-1 data
- about 6B filtered tokens from The Stack + StackOverflow, under 1B synthetic textbook tokens (GPT-3.5), about 180M tokens of synthetic exercises for fine-tuning
- phi-1.5
- same 1.3B architecture; 150B training tokens; about 20B synthetic tokens from 20K seed topics
- phi-2
- 2.7B; 1.4T tokens over a 250B-token synthetic + filtered-web mix; 96 A100s for 14 days; context 2048
- phi-3-mini
- 3.8B; 32 layers, , 32 heads, vocabulary 32,064 (Llama 2 tokenizer); 3.3T tokens; 4K context, 128K with LongRoPE; MMLU 69%
- phi-3-small / medium
- 7B / 14B, 4.8T tokens each; small uses a tiktoken vocabulary of 100,352 and alternates dense with blocksparse attention; MMLU 75% / 78%
- Phi-4
- 14B dense; about 10T tokens; 4K pretraining context extended to 16K in mid-training; mix about 40% synthetic, 15% web, 15% web rewrites, 20% code, 10% targeted acquired sources
Trained far past compute-optimal
- Chinchilla's parametric fit (Hoffmann et al. 2022, "Approach 3") of loss on MassiveText
- training FLOPs; fixing C traces an iso-FLOP curve over N
- minimum of L(N, C/6N); this fit gives 45 to 96 tokens per parameter over the Phi compute range, more than the 20:1 of Chinchilla's other two methods
- inference FLOPs per generated token, which is why a smaller N is worth extra training compute
Every Phi model sits to the left of the compute-optimal point. phi-3-mini trained on 868 tokens per parameter. For its compute, about FLOPs, Chinchilla's fit puts the optimum near a 12.8B model. On web data, the fit says choosing 3.8B costs 0.024 nats of loss. In exchange, each generated token costs 3.4× fewer FLOPs, and the model fits on a phone. Training is paid once and inference is paid on every token served. Llama 3 8B, trained on 15T tokens (about 1,900 per parameter), shows that this trade has become standard.
Phi's claim goes further: a curated token is worth more than a web token. You can ask how much more it must be worth to close the gap. If each Phi token counted as web tokens, the data term would become . The break-even is the value at which phi-3-mini reaches the compute-optimal loss: about 2.1. The fit was made on web text, so is a what-if, not a measurement. It does give a scale. The Phi results imply curated tokens worth a small multiple of web tokens, not orders of magnitude.
Worked example: the family against the fit
| model | N | D | D/N | C = 6ND | fit-optimal N | loss gap | q* |
|---|---|---|---|---|---|---|---|
| phi-1 | 1.3B | 50B | 38 | 3.90e20 | 1.2B | 0.0003 | 1.00 |
| phi-1.5 | 1.3B | 150B | 115 | 1.17e21 | 2.0B | 0.0050 | 1.06 |
| phi-2 | 2.7B | 1.4T | 519 | 2.27e22 | 7.5B | 0.0201 | 1.60 |
| phi-3-mini | 3.8B | 3.3T | 868 | 7.52e22 | 12.8B | 0.0241 | 2.10 |
| phi-3-small | 7B | 4.8T | 686 | 2.02e23 | 20.0B | 0.0153 | 1.66 |
| phi-3-medium | 14B | 4.8T | 343 | 4.03e23 | 27.4B | 0.0056 | 1.19 |
| Phi-4 | 14B | 10T | 714 | 8.40e23 | 38.2B | 0.0112 | 1.57 |
phi-3-mini by hand
- 1Compute: FLOPs, and .
- 220:1 rule: B on B tokens. This model would cost more per generated token than phi-3-mini.
- 3Chinchilla fit: and , so B and B, 76 tokens per parameter.
- 4Losses: and , a gap of 0.0241 nats.
- 5Break-even worth: solve . The data term must fall to , which gives .
1import numpy as np23rng = np.random.default_rng(0)45# A pool of 20,000 documents, each represented by a 32-d embedding (phi-1 used a pretrained6# code model's embeddings). The hidden "educational value" is unknown to us.7n_docs, dim = 20_000, 328X = rng.standard_normal((n_docs, dim))9w_true = rng.standard_normal(dim)10value = X @ w_true / np.sqrt(dim) + 0.5 * rng.standard_normal(n_docs) # hidden quality11good = value > 1.0 # "textbook quality"1213# 1) An LLM annotator labels a small random sample (phi-1: about 100K files labelled by GPT-4).14# Its judgements are imperfect: flip 10% of the labels.15idx = rng.choice(n_docs, 1_000, replace=False)16y = good[idx].astype(float)17flip = rng.random(len(idx)) < 0.1018y[flip] = 1 - y[flip]1920# 2) Train a cheap classifier on the labels (phi-1 used a random forest; logistic regression here).21w, b = np.zeros(dim), 0.022for _ in range(2_000):23 p = 1 / (1 + np.exp(-(X[idx] @ w + b)))24 g = p - y25 w -= 0.1 * (X[idx].T @ g / len(idx) + 1e-3 * w)26 b -= 0.1 * g.mean()2728# 3) Score the whole pool and keep the top 17% (phi-1 kept about 6B of 35B tokens).29score = X @ w + b30keep = score >= np.quantile(score, 1 - 0.17)3132print(f"good share: pool {good.mean():.3f}, kept {good[keep].mean():.3f}") # pool 0.155, kept 0.56633print(f"recall of good docs: {good[keep].sum() / good.sum():.3f}") # 0.62234print(f"mean hidden value: pool {value.mean():.3f}, kept {value[keep].mean():.3f}") # 0.006, 1.12935# 4) The price: with a fixed token budget, a 17% pool is repeated about 6x more often.36budget, pool = 50e9, 35e937print(f"passes over kept data for a 50B-token budget: {budget / (0.17 * pool):.1f}") # 8.41from transformers import AutoModelForCausalLM, AutoTokenizer2import torch34name = "microsoft/Phi-3-mini-4k-instruct"5tok = AutoTokenizer.from_pretrained(name)6model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16, device_map="auto")78messages = [{"role": "user", "content": "Write a Python function that checks whether a string is a palindrome."}]9inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)10out = model.generate(inputs, max_new_tokens=200, do_sample=False)11print(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))12print(sum(p.numel() for p in model.parameters()) / 1e9, "B parameters")Use when
- +Strong reasoning and coding scores for the parameter count, so cheap inference and on-device deployment
- +Shows that data selection and synthesis are first-class scaling levers alongside N and D
- +Filtering scales cheaply: the strong model labels only a small sample
- +Open weights (MIT licence for Phi-3 and Phi-4) at sizes that fit consumer hardware
Avoid when
- −Weaker factual knowledge than larger models: small N stores fewer facts, and filtering removes many
- −Benchmark-shaped training data raises contamination and overfitting-to-evaluation concerns
- −Depends on a stronger teacher model to label and generate, with that model's biases and errors
- −Synthetic data needs careful diversity engineering, or the model learns a narrow style
Check yourself
Why did phi-1 train a random-forest classifier instead of asking GPT-4 to rate every file?
Check yourself
phi-3-mini (3.8B) trained on 3.3T tokens. By Chinchilla's parametric fit, what does it gain and lose relative to a compute-optimal model at the same training FLOPs?
Remember
- ✓Phi (Gunasekar et al. 2023 to Abdin et al. 2024) bets on data quality: classifier-filtered web data plus LLM-written synthetic textbooks, in plain dense decoders of 1.3B to 14B.
- ✓phi-1: 1.3B, about 7B curated tokens seen about 8 times, 50.6% HumanEval. phi-2: 2.7B on 1.4T. phi-3-mini: 3.8B on 3.3T, 69% MMLU. Phi-4: 14B on about 10T, 40% synthetic.
- ✓The filter is distilled judgement: a strong LLM labels a small sample, and a cheap classifier on embeddings scores the whole pool.
- ✓Phi models train at 38 to 868 tokens per parameter, far beyond compute-optimal, trading a small predicted loss gap for 2–3× cheaper inference.
- ✓Under Chinchilla's fit, phi-3-mini's tokens would need to be worth web tokens to match the compute-optimal loss. Contamination is the main caveat for every data-quality claim.