Synthetic data generation

Strong models write training data (Self-Instruct, Alpaca's 52K pairs, Evol-Instruct) that must be filtered hard; training generation after generation on model output alone erases rare cases (model collapse) unless real data stays in.

8 min read801 wordscore0 viewsupdated 2026-10-21

Definition

Synthetic data generation is producing training examples with a model instead of collecting them from people: instructions and answers, dialogues, preference pairs, code with tests, or labelled examples for a classifier. The standard recipe for instruction data is Self-Instruct (Wang et al., "Self-Instruct: Aligning Language Models with Self-Generated Instructions", 2022, ACL 2023). Its main long-run risk, model collapse, was characterised by Shumailov et al. ("AI models collapse when trained on recursively generated data", Nature, 2024).

Human demonstrations are slow and expensive. InstructGPT used about 13K prompts with demonstrations written by contractors. A generator model can write tens of thousands of examples in hours for a few hundred dollars. That is why most open instruction-tuned models since 2023 were trained largely on synthetic data (see Instruction tuning and Knowledge distillation).

In a real pipeline, synthetic data sits between a task spec and Supervised fine-tuning (SFT). Seed examples and a generator produce candidates. Filters drop the bad ones. Then deduplication, decontamination and a quality pass prepare the final mix (see Dataset curation for fine-tuning). Most of the engineering goes into the filters.

Self-Instruct and Alpaca

Self-Instruct starts from 175 human-written seed tasks, each with one instruction and one instance. Every step samples 8 tasks for the prompt, 6 from the seeds and 2 from earlier generations, and asks GPT-3 for a ninth. The model then writes input–output instances. Classification tasks are generated output-first (label, then an input for it) so that labels stay balanced; other tasks are input-first. A new instruction joins the pool only if its ROUGE-L overlap with every existing instruction is below 0.7.

ROUGE-L(c,r)=2PRP+R,P=LCS(c,r)∣c∣,  R=LCS(c,r)∣r∣;keep c  ⟺  max⁡r∈poolROUGE-L(c,r)<0.7\text{ROUGE-L}(c,r)=\frac{2PR}{P+R},\quad P=\frac{\mathrm{LCS}(c,r)}{|c|},\;R=\frac{\mathrm{LCS}(c,r)}{|r|};\qquad \text{keep } c \iff \max_{r\in\text{pool}}\text{ROUGE-L}(c,r)<0.7
c, rc,\ r
candidate instruction and an existing pool instruction, as word sequences
LCS\mathrm{LCS}
length of the longest common subsequence (in order, gaps allowed)
0.70.7
Self-Instruct's novelty threshold; lower values force more diverse tasks but reject more
Self-Instruct (Wang et al., 2022): bootstrap tasks from 175 seedstask pool175 seed tasks(human-written)+ kept generatedtasksprompt the LLM8 example tasks:6 seed + 2 generatedthen 'Task 9:'→ new instructionnew task + instances"Write a short poem about the ocean."classification → output-firstotherwise → input-firstfilters1. ROUGE-L ≥ 0.7 with any pool task → drop vs "Write a poem about the sea": 0.769 → drop2. mentions image, picture, graph → drop3. empty, duplicated or invalid instances → dropkept → back into the pool; repeat to 52K tasksresult52K instructions82K instancesGPT-3 tuned on them≈ InstructGPT-001on SuperNIEvol-Instruct (WizardLM, Xu et al., 2023)in-depth: add constraints, deepen, concretize, more reasoning steps, complicate the inputin-breadth: mutate into a new, rarer taskSFT datasetAlpaca: 52K pairstext-davinci-003→ LLaMA 7B
fig.The Self-Instruct loop. A pool of 175 seed tasks supplies 8 in-context examples; the model writes a new task and its instances; filters reject near-duplicates (ROUGE-L ≥ 0.7), tasks a text model cannot do (images, pictures, graphs) and malformed instances; survivors join the pool. Evol-Instruct then rewrites instructions to be harder or broader. The output feeds SFT.

Stanford's Alpaca (Taori et al., 2023) reused the recipe with text-davinci-003 as the generator. It wrote 20 instructions per call and skipped the classification step. The result was 52K instruction–response pairs for under $500 of API calls. Fine-tuning LLaMA 7B on them took 3 hours on eight 80 GB A100s. Alpaca's licence was non-commercial, because of the LLaMA licence and the generator's terms of use.

Making data harder and better

Self-Instruct's tasks are short and easy. Evol-Instruct (Xu et al., "WizardLM", 2023) asks a model to rewrite each instruction. In-depth evolution adds constraints, deepens the question, makes it more concrete, adds reasoning steps or complicates the input. In-breadth evolution writes a new task on a rarer topic. A filter drops failed evolutions, such as copies of the original or responses that are only an apology. Four rounds starting from Alpaca's 52K gave about 250K instructions for WizardLM.

Where answers can be checked, use the check as the filter. Run generated code against generated unit tests. Keep chain-of-thought solutions only when the final answer matches a known one, as in STaR (Zelikman et al., 2022). Pick the best of several samples with a reward model or LLM-as-a-judge. Phi-1 (Gunasekar et al., "Textbooks Are All You Need", 2023) shows how far careful curation goes: a 1.3B code model trained on 6B tokens of filtered web code plus under 1B tokens of GPT-3.5-written textbooks, then fine-tuned on about 180M tokens of synthetic exercises, reached 50.6% pass@1 on HumanEval.

Reference numbers

Self-Instruct
175 seeds → 52K instructions, 82K instances from GPT-3. Tuned GPT-3 gained 33 points absolute on Super-NaturalInstructions, on par with InstructGPT-001.
Alpaca
52K pairs from text-davinci-003, under $500 to generate; LLaMA 7B fine-tuned in 3 h on 8 × A100 80 GB.
Evol-Instruct
4 evolution rounds from Alpaca's 52K → about 250K instructions; WizardLM is LLaMA fine-tuned on them.
Model collapse
Shumailov et al. fine-tuned OPT-125m on WikiText-2 over many generations. Perplexity on real data rose each generation; keeping 10% of the original data in every generation made the degradation much smaller.
Accumulation
Gerstgrasser et al. (2024): if each generation adds synthetic data to all previous data instead of replacing it, test error stays bounded rather than growing.

Model collapse

Suppose each model generation is trained only on samples from the previous one. Every finite sample under-represents rare events, and a model fitted to it gives them less probability, so the next sample has fewer still. Shumailov et al. found that the tails of the distribution go first (early collapse). Later the model converges to a narrow distribution unlike the original (late collapse). The simplest case shows the mechanism. Fit a Gaussian by maximum likelihood to nn samples of the previous fit:

E[σ^t2]=(n−1n)tσ2,Pr⁡[type i lost in one generation]=(1−qi)n\mathbb E\big[\hat\sigma^2_t\big]=\Big(\frac{n-1}{n}\Big)^t\sigma^2,\qquad \Pr[\text{type } i \text{ lost in one generation}]=(1-q_i)^n
σ^t2\hat\sigma^2_t
variance of the generation-tt fit; each MLE step shrinks it by (n−1)/n(n-1)/n in expectation
qiq_i
probability the current model gives to a rare type ii (a phrasing, a topic, a dialect)
nn
training samples per generation; a type with zero count gets probability 0 and never returns
generation 10: fit to 500 samples of generation 90.0010.010.1tail (ranks 31–60)p(type), log scale · line = original · red = losttype rank 1 … 60 (Zipf, s = 1.1)00.51010203040types kept / 60 · TV distancegenerationtypes kept25 / 60tail types lost25 / 30tail mass0.044 (orig. 0.119)TV(q_t, P0)0.217A type with probability q draws zerosamples with chance (1 − q)^n and isthen gone for every later generation.Fresh real data re-injects lost types;accumulating keeps the first sample.TV = ½ Σ |q_t(i) − P0(i)|; one seeded run, so curves are noisy at small n
training set
fig.Repeated maximum-likelihood fits of a 60-type Zipf distribution (s = 1.1). Defaults: 500 samples per generation, no real data, replacing the training set each time. Generation 1 keeps 57 of 60 types. By generation 10 only 25 remain; 25 of the 30 tail types are gone, tail mass is 0.044 instead of 0.119, and TV distance is 0.217. At generation 40, 12 types remain and TV is 0.419. With 10% fresh real data, 38 types survive to generation 40 (TV 0.263); with 20%, 41 survive (TV 0.211). Accumulating all data keeps 57 types and TV stays near 0.146.

Worked example

QuantityComputationResult
Candidate "Write a short poem about the ocean" vs "Write a poem about the sea"LCS = write a poem about the = 5; P = 5/7, R = 5/6; 2PR/(P+R)0.769 → rejected
"List three uses of baking soda" vs "Give three tips for staying healthy"LCS = three = 1; P = R = 1/60.167 → kept
Type rank 60 lost in one generationq = 0.0028, n = 500: (1 − 0.0028)^5000.246
Type rank 30 lost in one generationq = 0.0060: (1 − 0.0060)^5000.049
Gaussian variance after 10 generationsn = 10: (9/10)^100.349 σ²
Self-Instruct's ROUGE-L filter (threshold 0.7) and the collapse formulas, with the numbers from the figure

Rare types are lost far more often than common ones. Once lost, nothing in a closed loop can bring them back. That is why the tail goes first.

python
1import re
2
3def lcs(a, b):
4 """Length of the longest common subsequence of two token lists."""
5 prev = [0] * (len(b) + 1)
6 for x in a:
7 cur = [0]
8 for j, y in enumerate(b):
9 cur.append(prev[j] + 1 if x == y else max(prev[j + 1], cur[j]))
10 prev = cur
11 return prev[-1]
12
13def rouge_l(cand, ref):
14 c, r = cand.lower().split(), ref.lower().split()
15 m = lcs(c, r)
16 if m == 0:
17 return 0.0
18 p, rec = m / len(c), m / len(r)
19 return 2 * p * rec / (p + rec)
20
21BLOCK = re.compile(r"\b(image|images|picture|graph|graphs)\b", re.I)
22
23def keep(new, pool, threshold=0.7):
24 if BLOCK.search(new) or not 3 <= len(new.split()) <= 150:
25 return False
26 return max((rouge_l(new, old) for old in pool), default=0.0) < threshold
27
28pool = ["Write a poem about the sea", "Give three tips for staying healthy"]
29for cand in ["Write a short poem about the ocean",
30 "List three uses of baking soda",
31 "Describe the picture in one sentence"]:
32 print(f"{cand!r}: max ROUGE-L {max(rouge_l(cand, p) for p in pool):.3f}, keep={keep(cand, pool)}")
33# 'Write a short poem about the ocean': max ROUGE-L 0.769, keep=False
34# 'List three uses of baking soda': max ROUGE-L 0.167, keep=True
35# 'Describe the picture in one sentence': max ROUGE-L 0.167, keep=False
Self-Instruct-style novelty and keyword filters (pure Python, runs as is). The generator call is left out; in production, embedding similarity often replaces ROUGE-L for scale.

Use when

  • +Orders of magnitude cheaper and faster than human annotation.
  • +Targeted: generate exactly the tasks, formats and edge cases you lack.
  • +Verifiable domains (code, maths, SQL) allow automatic correctness filters.

Avoid when

  • −Bounded by the generator: errors, biases and blind spots carry over.
  • −Low diversity without explicit novelty filters; easy to overfit one style.
  • −Recursive training on model output loses the tails (model collapse); needs real data in the mix.

Check yourself

A team retrains its model every month only on text sampled from last month's model. What happens first?

Check yourself

Why does Self-Instruct generate classification instances output-first?

Remember

  • ✓Self-Instruct: 175 seeds, 8 in-context tasks (6 seed + 2 generated), novelty filter ROUGE-L < 0.7 → 52K instructions; Alpaca reused it with text-davinci-003 for under $500.
  • ✓Evol-Instruct rewrites instructions deeper (constraints, reasoning steps) or broader (new rare tasks), about 250K from Alpaca's 52K.
  • ✓Filter on correctness (tests, answer checks, judges) and diversity, not fluency; read samples.
  • ✓Model collapse: E[σ^t2]=((n−1)/n)tσ2\mathbb E[\hat\sigma^2_t]=((n-1)/n)^t\sigma^2 and a rare type is lost with chance (1−q)n(1-q)^n per generation, so tails vanish first.
  • ✓Keep real data in every generation, or accumulate rather than replace; both stop the collapse.
← back to Generative AI roadmap