# Phi

> Microsoft's Phi models (phi-1, 2023, to Phi-4, 2024) are small dense decoders of 1.3B to 14B parameters trained on classifier-filtered web data plus LLM-written synthetic textbooks, at 38 to 868 tokens per parameter instead of Chinchilla's 20, trading parameters for data quality.

Source: https://www.mlfoundry.in/learn/phi  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 15 min read · updated 2026-10-13

**Phi:** is Microsoft Research's series of small language models whose central bet is that **data quality can substitute for scale**. It began with phi-1 in Gunasekar et al., "Textbooks Are All You Need" (2023), a 1.3B-parameter Python model trained on "textbook-quality" data. That data was web code filtered by a classifier trained on GPT-4 quality labels, plus synthetic textbooks and exercises written by GPT-3.5. Later releases applied the recipe to general language: phi-1.5 (Li et al., 2023), phi-2 (2.7B, December 2023), the Phi-3 family (Abdin et al., 2024; phi-3-mini is 3.8B) and Phi-4 (14B, Abdin et al., December 2024). Architecturally they are ordinary decoder-only transformers. The differences are in the data and in how long they train.

Scaling laws had made model size the default lever. Kaplan et al. (2020) and then Hoffmann et al. (2022, "Chinchilla") described loss as a smooth function of parameters $N$ and tokens $D$, measured on web-scale text of roughly fixed quality. Chinchilla's popular takeaway was that a compute-optimal model should see about **20 tokens per parameter**. These laws treat a token as a token. The Phi papers asked how much the *distribution* of the tokens matters. If most web text is boilerplate, ads, or code without context, a model may spend most of its capacity on material that teaches little.

phi-1 was the test case. Its training corpus was about 7B tokens, and the model saw about 50B tokens in total, roughly 8 passes. On HumanEval it reached 50.6% pass@1, and 55.5% on MBPP. For comparison, StarCoder (15.5B parameters, 1T training tokens) scored 33.6%. A 350M-parameter sibling, phi-1-small, reached 45% on HumanEval. The result was surprising enough that much of the paper is spent on contamination checks.

## The recipe: filter, then synthesise

**Filtering.** The phi-1 authors took the deduplicated Python subset of The Stack and StackOverflow, about 35B tokens. They asked GPT-4 to rate about 100K samples by "educational value for a student whose goal is to learn basic coding concepts". They then trained a random-forest classifier on embeddings from a pretrained code model to predict that rating, and used it to score the whole pool. The kept portion was about 6B tokens. GPT-4 is far too expensive to label 35B tokens, so the trick is to use it as a *teacher for a cheap filter*.

**Synthesis.** They added about 1B tokens of GPT-3.5-generated Python "textbooks": explanations interleaved with code. For fine-tuning there was a separate set of about 180M tokens of exercises, function signatures with docstrings to complete. The hard part of synthetic data is *diversity*. An LLM asked a thousand times for "a Python textbook" writes nearly the same one. The authors injected randomness by constraining topics and target audiences. phi-1.5 scaled this up with about 20B tokens of synthetic text on common sense and general knowledge, seeded from 20K hand-picked topics.

$$
\text{keep}(x)=\mathbb 1\big[f_\theta(\mathrm{emb}(x))\ge\tau\big],\qquad \theta=\arg\min_\theta\sum_{x\in S}\ell\big(f_\theta(\mathrm{emb}(x)),\,y_{\text{LLM}}(x)\big),\qquad |S|\ll|\text{pool}|
$$

- $y_{\text{LLM}}(x)$: the strong model's judgement of educational value, obtained only for a small sample $S$
- $f_\theta$: a cheap classifier (random forest in phi-1) on frozen embeddings, run over the whole pool
- $\tau$: threshold chosen to keep a target fraction; lower τ keeps more tokens of lower average quality
- $\mathrm{emb}(x)$: a pretrained model's embedding of the document; phi-1 used a code language model

> Figure: The Phi data recipe and lineage. Top: web and code are scored by a cheap classifier trained on a small sample of LLM "educational value" labels, and only the top slice is kept. In parallel, an LLM writes textbooks, exercises and rewrites from seeded topics so the output stays diverse. Both feed a multi-epoch pretraining mix for a small decoder-only model, followed by SFT (phi-1 used about 180M tokens of synthetic exercises) and, from Phi-3 on, DPO. Bottom: five releases from phi-1 (1.3B, about 50B tokens seen) to Phi-4 (14B, about 10T tokens, 40% synthetic).

## From phi-1 to Phi-4

**phi-1.5** kept phi-1's 1.3B architecture and moved from code to general reasoning. It trained on 150B tokens: phi-1's 7B corpus plus about 20B new synthetic tokens. The report claims natural-language performance comparable to models 5× larger. Its variants with a 95B-token filtered web set showed that adding filtered web data on top of the synthetic set helped. **phi-2** (2.7B) was released through a blog post rather than a paper. It trained on 1.4T tokens, multiple passes over a 250B-token mixture of GPT-3.5 synthetic text and GPT-4-assessed web data from Falcon RefinedWeb and SlimPajama, on 96 A100s for 14 days. It was initialised by scaling up the knowledge of phi-1.5. It was a base model, without instruction tuning or RLHF.

**Phi-3** (April 2024) went from proof of concept to product. phi-3-mini has 3.8B parameters, 32 layers, $d=3072$ and the Llama 2 tokenizer. It trained on 3.3T tokens and scored 69% on MMLU and 8.38 on MT-Bench. Quantised to 4 bits it takes about 1.8 GB and runs on an iPhone 14 at more than 12 tokens per second. Training had two phases. Phase 1 was mostly filtered web text, for general knowledge. Phase 2 mixed even more heavily filtered web text with synthetic data aimed at reasoning. The report calls this a **data-optimal regime**. For a small model, it removes web content that adds facts but not reasoning. Its example is the result of a particular day's Premier League game. phi-3-small (7B) and phi-3-medium (14B) trained on 4.8T tokens and reached 75% and 78% on MMLU.

**Phi-4** (14B, December 2024) made synthetic data the core of pretraining. About 40% of the roughly 10T training tokens came from 50 types of synthetic datasets, about 400B unweighted tokens in total. They were built by multi-agent prompting, self-revision and instruction reversal, and seeded from curated web, books and code. On STEM question-answering benchmarks such as GPQA and MATH, it outperformed GPT-4o, its teacher for much of that data. To address contamination concerns, the authors also tested it on the November 2024 AMC-10/12 maths contests, which postdate its training data. Post-training used SFT and **Direct preference optimization (DPO)**, including DPO pairs built by "pivotal token search", which finds the single tokens that most change the probability of a correct answer.

**Configurations reported by the authors**

- **phi-1:** 1.3B; 24 layers, $d=2048$, 32 heads of 64, MLP 8192, rotary dimension 32, context 2048, FlashAttention, parallel attention/MLP; about 4 days on 8 A100s; HumanEval 50.6%, MBPP 55.5%
- **phi-1 data:** about 6B filtered tokens from The Stack + StackOverflow, under 1B synthetic textbook tokens (GPT-3.5), about 180M tokens of synthetic exercises for fine-tuning
- **phi-1.5:** same 1.3B architecture; 150B training tokens; about 20B synthetic tokens from 20K seed topics
- **phi-2:** 2.7B; 1.4T tokens over a 250B-token synthetic + filtered-web mix; 96 A100s for 14 days; context 2048
- **phi-3-mini:** 3.8B; 32 layers, $d=3072$, 32 heads, vocabulary 32,064 (Llama 2 tokenizer); 3.3T tokens; 4K context, 128K with LongRoPE; MMLU 69%
- **phi-3-small / medium:** 7B / 14B, 4.8T tokens each; small uses a tiktoken vocabulary of 100,352 and alternates dense with blocksparse attention; MMLU 75% / 78%
- **Phi-4:** 14B dense; about 10T tokens; 4K pretraining context extended to 16K in mid-training; mix about 40% synthetic, 15% web, 15% web rewrites, 20% code, 10% targeted acquired sources

## Trained far past compute-optimal

$$
L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}},\quad E=1.69,\ A=406.4,\ B=410.7,\ \alpha=0.34,\ \beta=0.28;\qquad N_{opt}(C)=G\Big(\frac C6\Big)^{\frac{\beta}{\alpha+\beta}},\ G=\Big(\frac{\alpha A}{\beta B}\Big)^{\frac1{\alpha+\beta}}
$$

- $L(N,D)$: Chinchilla's parametric fit (Hoffmann et al. 2022, "Approach 3") of loss on MassiveText
- $C\approx6ND$: training FLOPs; fixing C traces an iso-FLOP curve over N
- $N_{opt}$: minimum of L(N, C/6N); this fit gives 45 to 96 tokens per parameter over the Phi compute range, more than the 20:1 of Chinchilla's other two methods
- $2N$: inference FLOPs per generated token, which is why a smaller N is worth extra training compute

Every Phi model sits to the left of the compute-optimal point. phi-3-mini trained on 868 tokens per parameter. For its compute, about $7.5\times10^{22}$ FLOPs, Chinchilla's fit puts the optimum near a 12.8B model. On web data, the fit says choosing 3.8B costs 0.024 nats of loss. In exchange, each generated token costs 3.4× fewer FLOPs, and the model fits on a phone. Training is paid once and inference is paid on every token served. Llama 3 8B, trained on 15T tokens (about 1,900 per parameter), shows that this trade has become standard.

Phi's claim goes further: a curated token is worth more than a web token. You can ask how much more it must be worth to close the gap. If each Phi token counted as $q$ web tokens, the data term would become $B/(qD)^\beta$. The break-even $q^\ast$ is the value at which phi-3-mini reaches the compute-optimal loss: about 2.1. The fit was made on web text, so $q$ is a what-if, not a measurement. It does give a scale. The Phi results imply curated tokens worth a small multiple of web tokens, not orders of magnitude.

> Figure: Chinchilla's fit L(N, D) = 1.69 + 406.4/N^0.34 + 410.7/D^0.28 along the iso-FLOP curve of the chosen Phi model. Defaults: phi-3-mini (3.8B on 3.3T tokens) has 868 tokens per parameter and C = 7.52e22 FLOPs. At that compute the fit's optimum is 12.8B parameters on 977B tokens, and the 20:1 rule gives 25.0B on 501B. The fit predicts loss 2.0431 for phi-3-mini against 2.0191 at the optimum (+0.0241), for 7.6 instead of 25.7 GFLOPs per generated token (3.38×). The break-even token worth is q* = 2.10. Raising the slider to 2.0 brings phi-3-mini to 2.0205, just short of 2.0191. phi-1 (38 tokens per parameter) is nearly compute-optimal (+0.0003, q* = 1.00). phi-3-medium and Phi-4 are both 14B. Doubling the data from 4.8T to 10T moves q* from 1.19 to 1.57, and Phi-4's optimum would be a 38.2B model costing 2.73× more per token. The bars compare tokens per parameter across the family with the 20:1 line.

## Worked example: the family against the fit

Compute, compute-optimal size and break-even token worth (Chinchilla Approach 3 fit; computed in Python)

| model | N | D | D/N | C = 6ND | fit-optimal N | loss gap | q* |
| --- | --- | --- | --- | --- | --- | --- | --- |
| phi-1 | 1.3B | 50B | 38 | 3.90e20 | 1.2B | 0.0003 | 1.00 |
| phi-1.5 | 1.3B | 150B | 115 | 1.17e21 | 2.0B | 0.0050 | 1.06 |
| phi-2 | 2.7B | 1.4T | 519 | 2.27e22 | 7.5B | 0.0201 | 1.60 |
| phi-3-mini | 3.8B | 3.3T | 868 | 7.52e22 | 12.8B | 0.0241 | 2.10 |
| phi-3-small | 7B | 4.8T | 686 | 2.02e23 | 20.0B | 0.0153 | 1.66 |
| phi-3-medium | 14B | 4.8T | 343 | 4.03e23 | 27.4B | 0.0056 | 1.19 |
| Phi-4 | 14B | 10T | 714 | 8.40e23 | 38.2B | 0.0112 | 1.57 |

**phi-3-mini by hand**

1. Compute: $C=6ND=6\cdot3.8\times10^9\cdot3.3\times10^{12}=7.52\times10^{22}$ FLOPs, and $D/N=3.3\times10^{12}/3.8\times10^{9}=868$.
2. 20:1 rule: $N=\sqrt{C/120}=25.0$B on $20N=501$B tokens. This model would cost $6.6\times$ more per generated token than phi-3-mini.
3. Chinchilla fit: $\beta/(\alpha+\beta)=0.452$ and $G=1.345$, so $N_{opt}=1.345\cdot(C/6)^{0.452}=12.8$B and $D_{opt}=C/(6N_{opt})=977$B, 76 tokens per parameter.
4. Losses: $L(3.8\text B,3.3\text T)=1.69+0.2248+0.1283=2.0431$ and $L(12.8\text B,977\text B)=2.0191$, a gap of 0.0241 nats.
5. Break-even worth: solve $1.69+0.2248+410.7/(q\cdot3.3\times10^{12})^{0.28}=2.0191$. The data term must fall to $0.1043$, which gives $q^\ast=(410.7/0.1043)^{1/0.28}/3.3\times10^{12}=2.10$.

```python
import numpy as np

rng = np.random.default_rng(0)

# A pool of 20,000 documents, each represented by a 32-d embedding (phi-1 used a pretrained
# code model's embeddings). The hidden "educational value" is unknown to us.
n_docs, dim = 20_000, 32
X = rng.standard_normal((n_docs, dim))
w_true = rng.standard_normal(dim)
value = X @ w_true / np.sqrt(dim) + 0.5 * rng.standard_normal(n_docs)   # hidden quality
good = value > 1.0                                                      # "textbook quality"

# 1) An LLM annotator labels a small random sample (phi-1: about 100K files labelled by GPT-4).
#    Its judgements are imperfect: flip 10% of the labels.
idx = rng.choice(n_docs, 1_000, replace=False)
y = good[idx].astype(float)
flip = rng.random(len(idx)) < 0.10
y[flip] = 1 - y[flip]

# 2) Train a cheap classifier on the labels (phi-1 used a random forest; logistic regression here).
w, b = np.zeros(dim), 0.0
for _ in range(2_000):
    p = 1 / (1 + np.exp(-(X[idx] @ w + b)))
    g = p - y
    w -= 0.1 * (X[idx].T @ g / len(idx) + 1e-3 * w)
    b -= 0.1 * g.mean()

# 3) Score the whole pool and keep the top 17% (phi-1 kept about 6B of 35B tokens).
score = X @ w + b
keep = score >= np.quantile(score, 1 - 0.17)

print(f"good share: pool {good.mean():.3f}, kept {good[keep].mean():.3f}")     # pool 0.155, kept 0.566
print(f"recall of good docs: {good[keep].sum() / good.sum():.3f}")            # 0.622
print(f"mean hidden value: pool {value.mean():.3f}, kept {value[keep].mean():.3f}")  # 0.006, 1.129
# 4) The price: with a fixed token budget, a 17% pool is repeated about 6x more often.
budget, pool = 50e9, 35e9
print(f"passes over kept data for a 50B-token budget: {budget / (0.17 * pool):.1f}")  # 8.4
```

A phi-1-style quality filter on toy data: an LLM annotator labels a small sample, a cheap classifier learns its judgement, the pool is scored and the top 17% kept (numpy; run as shown)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

name = "microsoft/Phi-3-mini-4k-instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "Write a Python function that checks whether a string is a palindrome."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
print(sum(p.numel() for p in model.parameters()) / 1e9, "B parameters")
```

Using a released Phi model with Hugging Face transformers (not run here; needs torch and a model download)

> **Benchmarks can leak into curated data:** If the training set is designed to look like the test set, contamination becomes the main threat to validity. Textbook exercises look like HumanEval, and synthetic Q&A looks like MMLU. The phi-1 authors pruned exercises similar to HumanEval by embedding and syntax-tree similarity, retrained, and still beat larger models. They also graded 50 new, unusual problems. Phi-4 was tested on maths contests held after its data was collected. Treat any data-quality result as provisional until it holds on fresh, held-out tasks. Benchmark wins do not guarantee broad competence: small Phi models are weaker on factual recall (e.g. TriviaQA), as the Phi-3 report says.

> **Filtering is distillation of a judgement:** The expensive model never sees most of the pool. It labels a small sample. A cheap classifier learns that judgement and applies it at scale. The same pattern, a strong model's preferences compressed into a filter or into generated text, underlies most modern pretraining pipelines. Examples include FineWeb-Edu's educational-value classifier and the Llama 3 quality classifiers. In this sense Phi is **Knowledge distillation** through the data rather than through the logits.

> **Repetition is the hidden cost:** Filtering hard shrinks the pool, so a fixed token budget means more epochs. phi-1 kept about 6B of 35B tokens and trained about 8 passes. Muennighoff et al. ("Scaling Data-Constrained Language Models", 2023) found that up to about 4 epochs of repeated data are almost as good as fresh data, with returns diminishing quickly after that. The best threshold balances a higher average quality against the value lost to repetition. Synthetic data partly lifts this limit, because more of it can be generated.

**Strengths**

- Strong reasoning and coding scores for the parameter count, so cheap inference and on-device deployment
- Shows that data selection and synthesis are first-class scaling levers alongside N and D
- Filtering scales cheaply: the strong model labels only a small sample
- Open weights (MIT licence for Phi-3 and Phi-4) at sizes that fit consumer hardware

**Limitations**

- Weaker factual knowledge than larger models: small N stores fewer facts, and filtering removes many
- Benchmark-shaped training data raises contamination and overfitting-to-evaluation concerns
- Depends on a stronger teacher model to label and generate, with that model's biases and errors
- Synthetic data needs careful diversity engineering, or the model learns a narrow style

**Question:** Why did phi-1 train a random-forest classifier instead of asking GPT-4 to rate every file?

- A. GPT-4 cannot read Python
- B. Labelling a 35B-token pool with GPT-4 would be prohibitively expensive; GPT-4 labelled about 100K samples and a cheap classifier on embeddings extended that judgement to the whole pool
- C. Random forests are more accurate than GPT-4 at judging code
- D. The classifier also generated the synthetic textbooks

**Answer:** B. Labelling a 35B-token pool with GPT-4 would be prohibitively expensive; GPT-4 labelled about 100K samples and a cheap classifier on embeddings extended that judgement to the whole pool — The strong model is used as a teacher for a filter. Its labels on a small sample train a classifier that is cheap enough to score everything, and the top-scoring slice (about 6B of 35B tokens) is kept.

**Question:** phi-3-mini (3.8B) trained on 3.3T tokens. By Chinchilla's parametric fit, what does it gain and lose relative to a compute-optimal model at the same training FLOPs?

- A. It gains lower loss and loses nothing
- B. It loses about 0.024 nats of predicted loss on web data but is about 3.4× cheaper per generated token than the 12.8B optimum
- C. It is exactly compute-optimal
- D. It needs 3.4× more training FLOPs

**Answer:** B. It loses about 0.024 nats of predicted loss on web data but is about 3.4× cheaper per generated token than the 12.8B optimum — At $C=7.52\times10^{22}$ the fit's optimum is 12.8B on 977B tokens with loss 2.0191, against 2.0431 for phi-3-mini. Training FLOPs are equal by construction. Inference cost is $2N$ per token, so $12.8/3.8=3.38\times$. If curated tokens were worth about 2.1 web tokens, the gap would close.

**Key points**

- Phi (Gunasekar et al. 2023 to Abdin et al. 2024) bets on data quality: classifier-filtered web data plus LLM-written synthetic textbooks, in plain dense decoders of 1.3B to 14B.
- phi-1: 1.3B, about 7B curated tokens seen about 8 times, 50.6% HumanEval. phi-2: 2.7B on 1.4T. phi-3-mini: 3.8B on 3.3T, 69% MMLU. Phi-4: 14B on about 10T, 40% synthetic.
- The filter is distilled judgement: a strong LLM labels a small sample, and a cheap classifier on embeddings scores the whole pool.
- Phi models train at 38 to 868 tokens per parameter, far beyond compute-optimal, trading a small predicted loss gap for 2–3× cheaper inference.
- Under Chinchilla's fit, phi-3-mini's tokens would need to be worth $q^\ast\approx2.1$ web tokens to match the compute-optimal loss. Contamination is the main caveat for every data-quality claim.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
