Definition
Instruction tuning is multi-task fine-tuning in which every training example is written as a natural-language instruction plus input, and the target is the answer. The aim is generalisation to tasks never seen in training, specified only by their instruction. It was introduced by Wei et al. ("Finetuned Language Models Are Zero-Shot Learners", ICLR 2022), whose 137B model FLAN was tuned on 62 datasets rendered with templates. Sanh et al. (T0, "Multitask Prompted Training Enables Zero-Shot Task Generalization", ICLR 2022) did the same with an encoder–decoder. Chung et al. ("Scaling Instruction-Finetuned Language Models", 2022) scaled it to 1,836 tasks in Flan-T5 and Flan-PaLM.
GPT-3 showed that a large LM can do a task from a few examples in its prompt. Zero-shot, with only a description, it was much weaker. Wei et al. argued that one reason is that a pretraining corpus rarely contains text shaped like "here is an instruction, now do it". The fix is to teach that format directly. Take existing supervised NLP datasets, such as NLI, sentiment, QA, translation and summarisation, and rewrite each example as an instruction. Fine-tune on the mixture, then ask the model to do something it was never trained on.
Several earlier lines of work point the same way. T5 (Raffel et al., 2020) cast every task as text-to-text with a task prefix, but evaluated on the tasks it trained on. Natural Instructions (Mishra et al., 2022) collected 61 tasks with full human-written instructions, and Super-NaturalInstructions (Wang et al., 2022) grew it to 1,616 tasks of 76 types. Instruction tuning is the bridge between this multi-task NLP tradition and the chat-oriented Supervised fine-tuning (SFT) that followed. The objective is the same, a masked next-token loss on the target. What differs is the data: templated academic tasks, rather than open-ended demonstrations.
Templates and held-out clusters
FLAN wrote ten templates per dataset. Up to three of them "turn the task around", for example asking the model to write a hypothesis for a given label rather than classify it. Classification templates list the answer options, so the model sees which strings are valid outputs. Each example is rendered with a randomly chosen template, so the model meets many phrasings of the same task.
The evaluation protocol is what makes "zero-shot" meaningful. The 62 datasets are grouped into 12 task clusters. To score the model on a cluster, say NLI, a separate model is tuned on all the other clusters. No NLI dataset in any phrasing is seen in training. So the score measures transfer to a new kind of task, not to a new dataset of a familiar kind.
Objective and mixture
- dataset m in the mixture, from one of the training clusters
- templates for dataset ; is the instruction-formatted input
- target text (a label word, an answer or a summary); loss only on , as in Supervised fine-tuning (SFT)
- mixing rate: the probability that a training example comes from dataset m
- number of training examples in dataset m
- cap: datasets larger than count as size (Raffel et al.'s "examples-proportional mixing")
- temperature: is proportional, and larger flattens toward uniform
- total training examples drawn (steps × batch)
The mixture matters because dataset sizes span three orders of magnitude or more. SNLI has 550,152 training pairs and CB has 250. Mixing by raw size gives SNLI 37% of the training signal in the figure below and CB 0.02%. The cap evens this out. The price is that small datasets are repeated more often, and repetition beyond a few epochs means memorisation. FLAN capped each dataset at 30k examples and used examples-proportional mixing.
The main instruction-tuned models
- FLAN (2022)
- LaMDA-PT 137B decoder; 62 datasets, 12 clusters, 10 templates each; 30k steps, batch 8,192 tokens, Adafactor, lr ; input 1,024 and target 256 tokens. Zero-shot FLAN beat zero-shot GPT-3 175B on 20 of 25 datasets, and beat few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA and StoryCloze
- T0 (2022)
- LM-adapted T5 11B; prompts from the PromptSource collection (P3); held-out task types NLI, coreference, sentence completion and word-sense disambiguation; often beat models up to 16 times its size zero-shot
- Super-NaturalInstructions (2022)
- 1,616 tasks of 76 types with expert-written instructions; Tk-Instruct (T5) beat InstructGPT on the held-out tasks despite being much smaller
- Flan-T5 / Flan-PaLM (2022)
- 1,836 tasks from four mixtures (Muffin, T0-SF, NIV2 and chain-of-thought); T5 80M to 11B and PaLM 8B to 540B. Flan-PaLM 540B scored 75.2% on five-shot MMLU, using about 0.2% of the pretraining compute
- Self-Instruct (2023)
- 175 human-written seed tasks; GPT-3 generated 52K instructions with 82K instances, filtered by ROUGE-L < 0.7 against the existing pool; +33% absolute over vanilla GPT-3 on Super-NaturalInstructions, close to InstructGPT-001
- Alpaca (2023)
- LLaMA 7B on 52K instruction–response pairs generated by text-davinci-003 with a Self-Instruct pipeline; under $600 in total, 3 hours on eight 80 GB A100s
What scales: tasks, model size, templates
More task clusters help. In FLAN's ablation, adding clusters to the training mix steadily raised average zero-shot accuracy on the held-out clusters, with no sign of saturation at the largest mixture tried. Chung et al. later scaled the number of tasks from 9 to 89, 282, 682 and 1,836. The held-out gains kept coming but flattened: most of the improvement arrived by 282 tasks. Super-NaturalInstructions found that performance grows roughly log-linearly with the number of training tasks, while more instances per task saturate quickly. Diversity of tasks matters more than examples per task.
Scale is a precondition. FLAN tuned models of 422M, 2B, 8B, 68B and 137B parameters. At 68B and 137B, instruction tuning improved held-out tasks substantially. At 8B and below, it hurt held-out performance. The authors' reading is that a small model spends its capacity fitting the training tasks, while a large one can also learn the general skill of following instructions. Later work with better mixtures, such as Flan-T5, brought gains down to small models, so this threshold depends on the recipe.
Instructions matter. FLAN compared training with natural-language templates against training with only the dataset name, or with no instruction. Both alternatives did clearly worse on held-out clusters, so the gain is not just multi-task learning. T0 found that more templates per dataset improved median held-out accuracy and reduced the variance across prompts. The Flan Collection (Longpre et al., ICML 2023) showed that mixing zero-shot, few-shot and chain-of-thought templates helps in every evaluation setting. Training on a mix of zero-shot and few-shot prompts improved performance in both settings. Chung et al. found that leaving out the nine chain-of-thought datasets degraded reasoning ability.
Synthetic instructions: Self-Instruct and Alpaca
Templated academic datasets cover classification and short answers well, but not the open-ended requests users make. Writing thousands of new tasks by hand is expensive. Self-Instruct (Wang et al., ACL 2023) bootstraps them. It starts from 175 seed tasks and prompts GPT-3 with a few of them to propose new instructions. GPT-3 then generates inputs and outputs, and new tasks are kept only if they differ from every existing one (ROUGE-L below 0.7). Tuning GPT-3 on the resulting 52K instructions improved it by 33 points on Super-NaturalInstructions.
Alpaca (Taori et al., Stanford, 2023) applied the same pipeline with text-davinci-003 as the generator and LLaMA 7B as the student. For under $600 it produced a model that, in the authors' small study, behaved qualitatively like text-davinci-003 on many prompts. Alpaca started the wave of distilled open chat models. It also showed the limits: the student imitates the teacher's style much faster than its knowledge, and the data inherits the teacher's errors. In practice, this is Knowledge distillation through text.
Worked example: mixing four datasets
Four NLI-and-QA datasets with real training sizes, a budget of examples, and three mixing rules.
| dataset | e_m | proportional | cap K = 3,000 | epochs (K = 3,000) | T = 2, no cap |
|---|---|---|---|---|---|
| SNLI | 550,152 | 97.84% | 34.32% | 0.0125 | 82.00% |
| BoolQ | 9,427 | 1.68% | 34.32% | 0.728 | 10.73% |
| RTE | 2,490 | 0.44% | 28.49% | 2.29 | 5.52% |
| CB | 250 | 0.04% | 2.86% | 2.29 | 1.75% |
Computing the rates
- 1Proportional: , with . SNLI gets , and every dataset is covered for the same epochs. CB would contribute about 9 examples in total.
- 2Cap : the capped sizes are , with sum . SNLI and BoolQ get each, RTE and CB .
- 3Epochs under the cap: SNLI , BoolQ , and RTE and CB both . Every dataset below the cap is seen for the same number of epochs, .
- 4Temperature without a cap uses weights , giving . That is gentler than a cap: SNLI still dominates, and CB rises from 9 to 350 examples.
1import random2import numpy as np34# A template turns one example into (instruction-formatted input, target).5def nli_templates():6 opts = "OPTIONS:\n- yes\n- it is not possible to tell\n- no"7 return [8 lambda e: (f"Premise: {e['p']}\nHypothesis: {e['h']}\nDoes the premise entail the hypothesis?\n{opts}", e["y"]),9 lambda e: (f"{e['p']}\nBased on the sentence above, can we conclude that \"{e['h']}\"?\n{opts}", e["y"]),10 # "turn the task around": given the label, write the hypothesis11 lambda e: (f"Premise: {e['p']}\nWrite a hypothesis that the premise {'supports' if e['y'] == 'yes' else 'contradicts' if e['y'] == 'no' else 'neither supports nor rules out'}.", e["h"]),12 ]1314def sentiment_templates():15 return [16 lambda e: (f"Review: {e['x']}\nIs this review positive or negative?", e["y"]),17 lambda e: (f"\"{e['x']}\"\nHow would the author rate the movie: positive or negative?", e["y"]),18 ]1920rte = [{"p": "The cat sat on the warm mat all afternoon.", "h": "The cat was outside.", "y": "it is not possible to tell"}]21sst = [{"x": "A tender, funny and beautifully acted film.", "y": "positive"}]22datasets = [ # (name, cluster, train size, examples we hold in memory here, templates)23 ("SNLI", "NLI", 550_152, rte, nli_templates()),24 ("RTE", "NLI", 2_490, rte, nli_templates()),25 ("SST-2", "sentiment", 67_349, sst, sentiment_templates()),26]2728def mixing_rates(sizes, cap=3_000, temperature=1.0):29 """Examples-proportional mixing with a per-dataset cap (Raffel et al.), optionally temperature-scaled."""30 w = np.minimum(np.asarray(sizes, float), cap) ** (1.0 / temperature)31 return w / w.sum()3233def held_out(datasets, cluster):34 """FLAN-style evaluation split: train on every cluster except the one evaluated zero-shot."""35 return [d for d in datasets if d[1] != cluster], [d for d in datasets if d[1] == cluster]3637train, test = held_out(datasets, "NLI")38print([d[0] for d in train], [d[0] for d in test]) # ['SST-2'] ['SNLI', 'RTE']3940rates_all = mixing_rates([d[2] for d in datasets])41print({d[0]: round(float(r), 4) for d, r in zip(datasets, rates_all)}) # {'SNLI': 0.3534, 'RTE': 0.2933, 'SST-2': 0.3534}4243rng = random.Random(0)44def sample(datasets, rates, n):45 for _ in range(n):46 name, _, _, examples, templates = datasets[rng.choices(range(len(datasets)), weights=rates)[0]]47 template = rng.choice(templates) # a fresh phrasing every time the example is drawn48 yield name, template(rng.choice(examples))4950for name, (x, y) in sample(datasets, rates_all, 2):51 print(f"[{name}]\n{x}\n-> {y}\n")52# [SST-2]53# "A tender, funny and beautifully acted film."54# How would the author rate the movie: positive or negative?55# -> positive56#57# [SNLI]58# Premise: The cat sat on the warm mat all afternoon.59# Write a hypothesis that the premise neither supports nor rules out.60# -> The cat was outside.Use when
- +Cheap: Flan-PaLM's tuning used about 0.2% of its pretraining compute
- +Large zero-shot gains on unseen task types at sufficient scale (20 of 25 datasets over GPT-3 for FLAN)
- +Builds on existing labelled datasets; templates are cheap to write
- +Gives a better starting point for later task-specific fine-tuning (Flan-T5 converged faster than T5 in Longpre et al.)
Avoid when
- −Hurt held-out performance for FLAN models of 8B and below
- −Academic tasks have short targets; tuned models can become terse and template-bound on open-ended requests
- −Mixture design (caps, temperature, formats) changes results and is easy to get wrong
- −Synthetic instruction data (Self-Instruct, Alpaca) inherits the generator's errors and style
Check yourself
In FLAN's evaluation, what does it mean for NLI to be held out?
Check yourself
With examples-proportional mixing capped at K = 3,000 and N = 100,000 training examples, why is CB (250 examples) repeated 4.73 times, the same as RTE (2,490)?
Remember
- ✓Instruction tuning fine-tunes on many tasks phrased as natural-language instructions, so the model can do unseen tasks from an instruction alone.
- ✓FLAN: 137B, 62 datasets, 12 clusters, 10 templates each, evaluated on whole held-out clusters. It beat zero-shot GPT-3 on 20 of 25 datasets.
- ✓Mixing: . Caps stop huge datasets from dominating, but repeat small ones for epochs.
- ✓Gains grow with the number and diversity of tasks (Flan: 9 → 1,836 tasks, flattening after about 282) and with model size. FLAN at 8B and below got worse on held-out tasks.
- ✓Varied templates help: zero-shot, few-shot, chain of thought and inverted tasks. Self-Instruct and Alpaca replaced human-written tasks with model-generated ones.