Instruction tuning

Instruction tuning fine-tunes an LM on many tasks phrased as natural-language instructions, so it follows instructions for unseen tasks zero-shot. FLAN (137B, 62 datasets, 12 clusters) beat zero-shot GPT-3 on 20 of 25 tasks, and Flan-PaLM scaled this to 1,836 tasks.

15 min read1341 wordscore0 viewsupdated 2026-10-13

Definition

Instruction tuning is multi-task fine-tuning in which every training example is written as a natural-language instruction plus input, and the target is the answer. The aim is generalisation to tasks never seen in training, specified only by their instruction. It was introduced by Wei et al. ("Finetuned Language Models Are Zero-Shot Learners", ICLR 2022), whose 137B model FLAN was tuned on 62 datasets rendered with templates. Sanh et al. (T0, "Multitask Prompted Training Enables Zero-Shot Task Generalization", ICLR 2022) did the same with an encoder–decoder. Chung et al. ("Scaling Instruction-Finetuned Language Models", 2022) scaled it to 1,836 tasks in Flan-T5 and Flan-PaLM.

GPT-3 showed that a large LM can do a task from a few examples in its prompt. Zero-shot, with only a description, it was much weaker. Wei et al. argued that one reason is that a pretraining corpus rarely contains text shaped like "here is an instruction, now do it". The fix is to teach that format directly. Take existing supervised NLP datasets, such as NLI, sentiment, QA, translation and summarisation, and rewrite each example as an instruction. Fine-tune on the mixture, then ask the model to do something it was never trained on.

Several earlier lines of work point the same way. T5 (Raffel et al., 2020) cast every task as text-to-text with a task prefix, but evaluated on the tasks it trained on. Natural Instructions (Mishra et al., 2022) collected 61 tasks with full human-written instructions, and Super-NaturalInstructions (Wang et al., 2022) grew it to 1,616 tasks of 76 types. Instruction tuning is the bridge between this multi-task NLP tradition and the chat-oriented Supervised fine-tuning (SFT) that followed. The objective is the same, a masked next-token loss on the target. What differs is the data: templated academic tasks, rather than open-ended demonstrations.

Templates and held-out clusters

FLAN wrote ten templates per dataset. Up to three of them "turn the task around", for example asking the model to write a hypothesis for a given label rather than classify it. Classification templates list the answer options, so the model sees which strings are valid outputs. Each example is rendered with a randomly chosen template, so the model meets many phrasings of the same task.

The evaluation protocol is what makes "zero-shot" meaningful. The 62 datasets are grouped into 12 task clusters. To score the model on a cluster, say NLI, a separate model is tuned on all the other clusters. No NLI dataset in any phrasing is seen in training. So the score measures transfer to a new kind of task, not to a new dataset of a familiar kind.

FLAN: 62 datasets in 12 task clusters · click a cluster to hold it outNLIcommonsensesentimentparaphraseclosed-book QAstruct to texttranslationreading comp.RC w/ commonsensecoreferencemisc.summarizationtrain: 11 clustersall but the dashed onetemplates10 per datasetmixturecapped, prop.fine-tuneLaMDA-PT 137Bzero-shot evalon NLIthe model never sees any dataset of the held-out cluster,so the score measures transfer to an unseen kind of taskpretrain → fine-tunee.g. BERT, T5fine-tune on task Ainfer on task Aone model per taskpromptinge.g. GPT-3no weight updatesfew-shot exemplarsin the contextinstruction tuninge.g. FLAN, T0fine-tune on B, C, Das instructionsinfer on unseen A
fig.FLAN's protocol. The 62 datasets fall into 12 task clusters. One cluster is held out (click to change it), and the model is tuned on the rest: each example is rendered with one of 10 templates and drawn from a capped mixture. It is then evaluated zero-shot on the held-out cluster. Below: instruction tuning against the two earlier paradigms. Pretrain-then-fine-tune needs one model per task. Prompting needs no training but relies on few-shot exemplars. Instruction tuning trains on many tasks and infers on an unseen one.

Objective and mixture

L(θ)=−∑mrm  E(x,y)∼Dm  Eτ∼Tm  log⁡pθ(y∣τ(x))\mathcal L(\theta)=-\sum_{m}r_m\;\mathbb E_{(x,y)\sim D_m}\;\mathbb E_{\tau\sim\mathcal T_m}\;\log p_\theta\big(y\mid\tau(x)\big)
DmD_m
dataset m in the mixture, from one of the training clusters
Tm\mathcal T_m
templates for dataset mm; τ(x)\tau(x) is the instruction-formatted input
yy
target text (a label word, an answer or a summary); loss only on yy, as in Supervised fine-tuning (SFT)
rmr_m
mixing rate: the probability that a training example comes from dataset m
rm=min⁡(em,K)1/T∑nmin⁡(en,K)1/Tepochs over Dm=rmNemr_m=\frac{\min(e_m,K)^{1/T}}{\sum_n\min(e_n,K)^{1/T}}\qquad\text{epochs over }D_m=\frac{r_m N}{e_m}
eme_m
number of training examples in dataset m
KK
cap: datasets larger than KK count as size KK (Raffel et al.'s "examples-proportional mixing")
TT
temperature: T=1T=1 is proportional, and larger TT flattens toward uniform
NN
total training examples drawn (steps × batch)

The mixture matters because dataset sizes span three orders of magnitude or more. SNLI has 550,152 training pairs and CB has 250. Mixing by raw size gives SNLI 37% of the training signal in the figure below and CB 0.02%. The cap evens this out. The price is that small datasets are repeated more often, and repetition beyond a few epochs means memorisation. FLAN capped each dataset at 30k examples and used examples-proportional mixing.

The main instruction-tuned models

FLAN (2022)
LaMDA-PT 137B decoder; 62 datasets, 12 clusters, 10 templates each; 30k steps, batch 8,192 tokens, Adafactor, lr 3⋅10−53\cdot10^{-5}; input 1,024 and target 256 tokens. Zero-shot FLAN beat zero-shot GPT-3 175B on 20 of 25 datasets, and beat few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA and StoryCloze
T0 (2022)
LM-adapted T5 11B; prompts from the PromptSource collection (P3); held-out task types NLI, coreference, sentence completion and word-sense disambiguation; often beat models up to 16 times its size zero-shot
Super-NaturalInstructions (2022)
1,616 tasks of 76 types with expert-written instructions; Tk-Instruct (T5) beat InstructGPT on the held-out tasks despite being much smaller
Flan-T5 / Flan-PaLM (2022)
1,836 tasks from four mixtures (Muffin, T0-SF, NIV2 and chain-of-thought); T5 80M to 11B and PaLM 8B to 540B. Flan-PaLM 540B scored 75.2% on five-shot MMLU, using about 0.2% of the pretraining compute
Self-Instruct (2023)
175 human-written seed tasks; GPT-3 generated 52K instructions with 82K instances, filtered by ROUGE-L < 0.7 against the existing pool; +33% absolute over vanilla GPT-3 on Super-NaturalInstructions, close to InstructGPT-001
Alpaca (2023)
LLaMA 7B on 52K instruction–response pairs generated by text-davinci-003 with a Self-Instruct pipeline; under $600 in total, 3 hours on eight 80 GB A100s
r_m ∝ min(e_m, K)^(1/T) · K = 3,000 · T = 1 · budget N = 100,000 examplesdatasetclustertrain sizemixing rateepochs seenSNLINLI550,15214.19%0.026 barely usedMNLINLI392,70214.19%0.036 barely usedQQPparaphrase363,84614.19%0.039 barely usedSQuAD v1.1reading comp.87,59914.19%0.16SST-2sentiment67,34914.19%0.21BoolQreading comp.9,42714.19%1.51RTENLI2,49011.78%4.73COPAcommonsense4001.89%4.73CBNLI2501.18%4.73share of training examples by task clusterNLI 41%reading comp. 28%paraphrase 14.2% · sentiment 14.2% · commonsense 1.9%one RTE-style example, template “zero-shot A” (illustrative, FLAN style)Premise: The cat sat on the warm mat all afternoon.Hypothesis: The cat was outside.Does the premise entail the hypothesis?OPTIONS: - yes - it is not possible to tell - no→ it is not possible to tell
template
fig.Nine real datasets with their official training-set sizes (1,474,215 examples in total), mixed by r_m ∝ min(e_m, K)^(1/T). The right column is how many epochs of each dataset a budget of N training examples covers. At the default (K = 3,000, T = 1, N = 100,000), the six datasets above the cap each get 14.19%, RTE 11.78%, COPA 1.89% and CB 1.18%. SNLI is then seen for only 0.026 of an epoch, while RTE, COPA and CB are all repeated 4.73 times. NLI takes 41.3% of the mixture. With no cap, every example is equally likely: each dataset gets the same 0.068 epochs, SNLI alone takes 37.32% and NLI takes 64.1%, but CB is 0.02% and commonsense 0.0%. A cap of 250 makes the mixture uniform (11.11% each) and repeats CB 44.4 times. Raising the temperature to 8 at K = 3,000 lifts CB to 8.64% and 34.5 epochs. With K = 3,000 and N = 1,000,000, BoolQ is repeated 15.1 times and RTE, COPA and CB 47.3 times: a recipe for memorising small datasets. Below the mixture, one RTE-style example is rendered with five FLAN-style templates: two zero-shot phrasings with an options suffix, few-shot, chain of thought, and an inverted task that generates the hypothesis from the label.

What scales: tasks, model size, templates

More task clusters help. In FLAN's ablation, adding clusters to the training mix steadily raised average zero-shot accuracy on the held-out clusters, with no sign of saturation at the largest mixture tried. Chung et al. later scaled the number of tasks from 9 to 89, 282, 682 and 1,836. The held-out gains kept coming but flattened: most of the improvement arrived by 282 tasks. Super-NaturalInstructions found that performance grows roughly log-linearly with the number of training tasks, while more instances per task saturate quickly. Diversity of tasks matters more than examples per task.

Scale is a precondition. FLAN tuned models of 422M, 2B, 8B, 68B and 137B parameters. At 68B and 137B, instruction tuning improved held-out tasks substantially. At 8B and below, it hurt held-out performance. The authors' reading is that a small model spends its capacity fitting the training tasks, while a large one can also learn the general skill of following instructions. Later work with better mixtures, such as Flan-T5, brought gains down to small models, so this threshold depends on the recipe.

Instructions matter. FLAN compared training with natural-language templates against training with only the dataset name, or with no instruction. Both alternatives did clearly worse on held-out clusters, so the gain is not just multi-task learning. T0 found that more templates per dataset improved median held-out accuracy and reduced the variance across prompts. The Flan Collection (Longpre et al., ICML 2023) showed that mixing zero-shot, few-shot and chain-of-thought templates helps in every evaluation setting. Training on a mix of zero-shot and few-shot prompts improved performance in both settings. Chung et al. found that leaving out the nine chain-of-thought datasets degraded reasoning ability.

Synthetic instructions: Self-Instruct and Alpaca

Templated academic datasets cover classification and short answers well, but not the open-ended requests users make. Writing thousands of new tasks by hand is expensive. Self-Instruct (Wang et al., ACL 2023) bootstraps them. It starts from 175 seed tasks and prompts GPT-3 with a few of them to propose new instructions. GPT-3 then generates inputs and outputs, and new tasks are kept only if they differ from every existing one (ROUGE-L below 0.7). Tuning GPT-3 on the resulting 52K instructions improved it by 33 points on Super-NaturalInstructions.

Alpaca (Taori et al., Stanford, 2023) applied the same pipeline with text-davinci-003 as the generator and LLaMA 7B as the student. For under $600 it produced a model that, in the authors' small study, behaved qualitatively like text-davinci-003 on many prompts. Alpaca started the wave of distilled open chat models. It also showed the limits: the student imitates the teacher's style much faster than its knowledge, and the data inherits the teacher's errors. In practice, this is Knowledge distillation through text.

Worked example: mixing four datasets

Four NLI-and-QA datasets with real training sizes, a budget of N=20,000N=20{,}000 examples, and three mixing rules.

datasete_mproportionalcap K = 3,000epochs (K = 3,000)T = 2, no cap
SNLI550,15297.84%34.32%0.012582.00%
BoolQ9,4271.68%34.32%0.72810.73%
RTE2,4900.44%28.49%2.295.52%
CB2500.04%2.86%2.291.75%
Mixing rates and epochs covered for N = 20,000 (computed in Python)

Computing the rates

  1. 1Proportional: rm=em/∑enr_m=e_m/\sum e_n, with ∑en=562,319\sum e_n=562{,}319. SNLI gets 550,152/562,319=97.84%550{,}152/562{,}319=97.84\%, and every dataset is covered for the same 20,000/562,319=0.035620{,}000/562{,}319=0.0356 epochs. CB would contribute about 9 examples in total.
  2. 2Cap K=3,000K=3{,}000: the capped sizes are 3000,3000,2490,2503000,3000,2490,250, with sum 8,7408{,}740. SNLI and BoolQ get 3000/8740=34.32%3000/8740=34.32\% each, RTE 28.49%28.49\% and CB 2.86%2.86\%.
  3. 3Epochs under the cap: SNLI 0.3432⋅20000/550152=0.01250.3432\cdot20000/550152=0.0125, BoolQ 0.7280.728, and RTE and CB both 2.292.29. Every dataset below the cap is seen for the same number of epochs, N/∑min⁡(en,K)N/\sum\min(e_n,K).
  4. 4Temperature T=2T=2 without a cap uses weights em=741.7,97.1,49.9,15.8\sqrt{e_m}=741.7,97.1,49.9,15.8, giving 82.0%,10.7%,5.5%,1.75%82.0\%,10.7\%,5.5\%,1.75\%. That is gentler than a cap: SNLI still dominates, and CB rises from 9 to 350 examples.
python
1import random
2import numpy as np
3
4# A template turns one example into (instruction-formatted input, target).
5def nli_templates():
6 opts = "OPTIONS:\n- yes\n- it is not possible to tell\n- no"
7 return [
8 lambda e: (f"Premise: {e['p']}\nHypothesis: {e['h']}\nDoes the premise entail the hypothesis?\n{opts}", e["y"]),
9 lambda e: (f"{e['p']}\nBased on the sentence above, can we conclude that \"{e['h']}\"?\n{opts}", e["y"]),
10 # "turn the task around": given the label, write the hypothesis
11 lambda e: (f"Premise: {e['p']}\nWrite a hypothesis that the premise {'supports' if e['y'] == 'yes' else 'contradicts' if e['y'] == 'no' else 'neither supports nor rules out'}.", e["h"]),
12 ]
13
14def sentiment_templates():
15 return [
16 lambda e: (f"Review: {e['x']}\nIs this review positive or negative?", e["y"]),
17 lambda e: (f"\"{e['x']}\"\nHow would the author rate the movie: positive or negative?", e["y"]),
18 ]
19
20rte = [{"p": "The cat sat on the warm mat all afternoon.", "h": "The cat was outside.", "y": "it is not possible to tell"}]
21sst = [{"x": "A tender, funny and beautifully acted film.", "y": "positive"}]
22datasets = [ # (name, cluster, train size, examples we hold in memory here, templates)
23 ("SNLI", "NLI", 550_152, rte, nli_templates()),
24 ("RTE", "NLI", 2_490, rte, nli_templates()),
25 ("SST-2", "sentiment", 67_349, sst, sentiment_templates()),
26]
27
28def mixing_rates(sizes, cap=3_000, temperature=1.0):
29 """Examples-proportional mixing with a per-dataset cap (Raffel et al.), optionally temperature-scaled."""
30 w = np.minimum(np.asarray(sizes, float), cap) ** (1.0 / temperature)
31 return w / w.sum()
32
33def held_out(datasets, cluster):
34 """FLAN-style evaluation split: train on every cluster except the one evaluated zero-shot."""
35 return [d for d in datasets if d[1] != cluster], [d for d in datasets if d[1] == cluster]
36
37train, test = held_out(datasets, "NLI")
38print([d[0] for d in train], [d[0] for d in test]) # ['SST-2'] ['SNLI', 'RTE']
39
40rates_all = mixing_rates([d[2] for d in datasets])
41print({d[0]: round(float(r), 4) for d, r in zip(datasets, rates_all)}) # {'SNLI': 0.3534, 'RTE': 0.2933, 'SST-2': 0.3534}
42
43rng = random.Random(0)
44def sample(datasets, rates, n):
45 for _ in range(n):
46 name, _, _, examples, templates = datasets[rng.choices(range(len(datasets)), weights=rates)[0]]
47 template = rng.choice(templates) # a fresh phrasing every time the example is drawn
48 yield name, template(rng.choice(examples))
49
50for name, (x, y) in sample(datasets, rates_all, 2):
51 print(f"[{name}]\n{x}\n-> {y}\n")
52# [SST-2]
53# "A tender, funny and beautifully acted film."
54# How would the author rate the movie: positive or negative?
55# -> positive
56#
57# [SNLI]
58# Premise: The cat sat on the warm mat all afternoon.
59# Write a hypothesis that the premise neither supports nor rules out.
60# -> The cat was outside.
Instruction-tuning data: templates, a FLAN-style held-out split, capped mixing, and sampling a fresh template per draw (runs with numpy; outputs shown)

Use when

  • +Cheap: Flan-PaLM's tuning used about 0.2% of its pretraining compute
  • +Large zero-shot gains on unseen task types at sufficient scale (20 of 25 datasets over GPT-3 for FLAN)
  • +Builds on existing labelled datasets; templates are cheap to write
  • +Gives a better starting point for later task-specific fine-tuning (Flan-T5 converged faster than T5 in Longpre et al.)

Avoid when

  • −Hurt held-out performance for FLAN models of 8B and below
  • −Academic tasks have short targets; tuned models can become terse and template-bound on open-ended requests
  • −Mixture design (caps, temperature, formats) changes results and is easy to get wrong
  • −Synthetic instruction data (Self-Instruct, Alpaca) inherits the generator's errors and style

Check yourself

In FLAN's evaluation, what does it mean for NLI to be held out?

Check yourself

With examples-proportional mixing capped at K = 3,000 and N = 100,000 training examples, why is CB (250 examples) repeated 4.73 times, the same as RTE (2,490)?

Remember

  • ✓Instruction tuning fine-tunes on many tasks phrased as natural-language instructions, so the model can do unseen tasks from an instruction alone.
  • ✓FLAN: 137B, 62 datasets, 12 clusters, 10 templates each, evaluated on whole held-out clusters. It beat zero-shot GPT-3 on 20 of 25 datasets.
  • ✓Mixing: rm∝min⁡(em,K)1/Tr_m\propto\min(e_m,K)^{1/T}. Caps stop huge datasets from dominating, but repeat small ones for N/∑min⁡(en,K)N/\sum\min(e_n,K) epochs.
  • ✓Gains grow with the number and diversity of tasks (Flan: 9 → 1,836 tasks, flattening after about 282) and with model size. FLAN at 8B and below got worse on held-out tasks.
  • ✓Varied templates help: zero-shot, few-shot, chain of thought and inverted tasks. Self-Instruct and Alpaca replaced human-written tasks with model-generated ones.
← back to Deep Learning roadmap