Definition
Supervised fine-tuning (SFT) is the stage that turns a pretrained next-token predictor into an assistant. You continue training on demonstrations: pairs of a prompt (the system message, user turns and any earlier context) and a target response written by a human or a stronger model. The loss is the usual causal language-modelling cross-entropy, computed only on the response tokens. The modern recipe comes from InstructGPT (Ouyang et al., "Training language models to follow instructions with human feedback", NeurIPS 2022), where SFT on about 13k labeller demonstrations was step 1, before reward modelling and RLHF. Llama 2-Chat (Touvron et al., 2023) and LIMA (Zhou et al., "LIMA: Less Is More for Alignment", NeurIPS 2023) are the reference points for data scale and quality.
A pretrained model continues documents. Ask GPT-3 "Explain the moon landing to a 6 year old" and it may continue with more questions like it, as if it were writing a list of prompts. It has the knowledge but not the behaviour. Few-shot prompting can coax out the behaviour, but it is brittle and uses up context. Pretraining and continued pretraining changes what the model knows. SFT changes the format of what it does: answer in the assistant role, follow the instruction, stop when done.
The name predates chat models. GPT-1 (Radford et al., 2018) already called its second stage "supervised fine-tuning", with a task head trained on labelled data. Instruction tuning (FLAN, T0, 2021–22) fine-tuned on many NLP tasks phrased as instructions, to get zero-shot transfer. InstructGPT fine-tuned on open-ended demonstrations of what users actually asked an API. The 1.3B InstructGPT model, after the full SFT + RLHF pipeline, was preferred by labellers over the 175B GPT-3. Today "SFT" usually means this chat-demonstration stage, run before preference tuning with RLHF or Direct preference optimization (DPO).
Objective and loss masking
- the prompt: system message, user turns, earlier assistant turns and template tokens. It is context, not a target
- the assistant response including its end-of-turn token, so the model learns when to stop
- normalise by the number of target tokens in the batch (token mean), so long answers weigh more than short ones
In code, the mask is the label value , which PyTorch's cross_entropy ignores by default. Inputs are the full rendered conversation, and labels are a copy with every prompt and template position set to . The model shifts labels by one internally, so position is scored on token . Llama 2-Chat describes it plainly: prompt and answer are concatenated with a special separator, and the loss on prompt tokens is zeroed out.
Why mask the prompt? User text is hard to predict. Learning to predict it spends capacity on imitating users, not on answering them. It also dominates the loss: in the worked example below, 5 prompt and template tokens contribute 92.7% of the unmasked loss sum. Masking is a default, not a law. Shi et al. ("Instruction Tuning With Loss Over Instructions", 2024) found that keeping the loss on instructions can help when responses are short and data is scarce.
Chat templates
A conversation is a list of messages, [{"role": "user", "content": ...}, {"role": "assistant", ...}]. The model sees one token sequence. A chat template serialises roles with special tokens. ChatML (used by Qwen and others) writes <|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n...<|im_end|>. Llama 2 uses [INST] <<SYS>>...<</SYS>> ... [/INST], and Llama 3 uses <|start_header_id|>user<|end_header_id|>\n\n...<|eot_id|>. In Hugging Face the template is a Jinja string stored with the tokenizer, rendered by tokenizer.apply_chat_template. With add_generation_prompt=True it appends the assistant header, so generation starts exactly where training targets started.
Packing and attention masking
SFT examples vary wildly in length: a one-line answer, then a 2,000-token code review. Padding every example to the maximum length wastes most of the compute on pad tokens. Packing concatenates several examples into each row of length . A causal mask alone then lets example attend to examples in the same row. This is cross-contamination: the model learns to condition on unrelated text, and positional encodings no longer start at 0. Krell et al. ("Efficient Sequence Packing without Cross-contamination", 2021) showed that block-diagonal masks plus per-example position ids give packed training the same result as unpacked training.
- additive attention mask for one packed row: causal and block-diagonal
- position id restarts at each example, so RoPE sees the same positions as in an unpacked batch
- in practice nobody builds : FlashAttention's variable-length kernels take cumulative sequence lengths and skip the off-diagonal blocks entirely (see FlashAttention)
Also set the label of each example's first token to . Otherwise the last token of one example is trained to predict the first token of the next. Hugging Face's DataCollatorWithFlattening does both: it concatenates examples, resets position_ids, and inserts a separator label.
Hyperparameters
SFT is a small nudge on a large model. Learning rates are roughly an order of magnitude below pretraining peaks: to for full fine-tuning of 7B–70B models, with warmup and cosine or linear decay. LoRA runs typically use to because only the small adapters move. Two or three epochs are typical. More epochs drive training loss toward zero on a few thousand examples, and the model starts reciting them.
Published SFT recipes
- InstructGPT (2022)
- GPT-3 base models, about 13k training prompts with labeller demonstrations; 16 epochs, cosine decay, residual dropout 0.2. SFT models overfit on validation loss after 1 epoch, but more epochs still improved reward-model score and human preference
- Alpaca (2023)
- LLaMA 7B, 52K demonstrations generated by text-davinci-003; 3 epochs, learning rate , batch 128, max length 512
- LIMA (2023)
- LLaMA 65B, 1,000 curated examples (750 from Stack Exchange, wikiHow and Reddit, 250 written by the authors); 15 epochs, AdamW , weight decay 0.1, lr decaying linearly to , batch 32, max 2048 tokens, residual dropout rising from 0.0 to 0.3 across layers
- Llama 2-Chat (2023)
- 27,540 vendor-written SFT annotations; cosine schedule, lr , weight decay 0.1, batch 64, sequence length 4096, 2 epochs; prompt loss zeroed
Data quality over quantity
LIMA fine-tuned LLaMA 65B on 1,000 carefully chosen examples, with no RLHF. In a human study, its answers were equivalent to or preferred over GPT-4's in 43% of cases. The authors proposed the Superficial Alignment Hypothesis: almost all knowledge is learned in pretraining, and SFT mainly teaches the format and style in which to present it. Llama 2's authors reached a similar conclusion. Setting aside millions of third-party examples in favour of fewer, higher-quality vendor annotations improved results notably, and they stopped at 27,540.
Quality here means correct, helpful answers in a consistent voice, diverse prompts, and no near-duplicates. LIMA's ablations found that prompt diversity and response quality mattered more than doubling quantity. A single bad pattern repeated across thousands of examples, such as a canned refusal or a stock opening, gets copied faithfully by the model.
Overfitting and forgetting
Validation loss on held-out demonstrations is a weak guide. InstructGPT and LIMA both trained well past the point where it started rising, and chose checkpoints by human or reward-model judgement instead. LIMA selected checkpoints between epochs 5 and 10 on a 50-example development set. Generation quality, not perplexity, is the target.
Catastrophic forgetting is the other risk. A narrow SFT set, such as only code or only one language, can erode general abilities that the base model had. The mitigations are a low learning rate, few epochs, a broad data mixture (often with some general chat or pretraining-style data mixed in), or a parameter-efficient method. Biderman et al. ("LoRA Learns Less and Forgets Less", 2024) measured exactly that trade-off: LoRA learned a new domain less fully than full fine-tuning, but kept more of the base model's other skills. See Parameter-efficient fine-tuning (PEFT).
Worked example: what the mask changes
One short exchange, "What is 2+2?" → "4.", with [A] the assistant header and [E] the end-of-turn token. Each row is the probability the model assigns to that token as the next one.
| target token | p(token | prefix) | −log p | label |
|---|---|---|---|
| What | 0.02 | 3.912 | −100 |
| is | 0.30 | 1.204 | −100 |
| 2+2 | 0.01 | 4.605 | −100 |
| ? | 0.40 | 0.916 | −100 |
| [A] | 0.95 | 0.051 | −100 |
| 4 | 0.90 | 0.105 | trained |
| . | 0.60 | 0.511 | trained |
| [E] | 0.80 | 0.223 | trained |
Computing the loss
- 1Masked SFT loss: .
- 2Unmasked loss over all 8 targets: . The prompt tokens supply of the sum, so the gradient would mostly teach the model to predict user questions.
- 3Batch it with a second example that has 30 response tokens, each with NLL . The token mean is , and the short example gets of the weight.
- 4A per-example mean, , would give each example half. Both are used. Pick one deliberately, and with gradient accumulation normalise by the target count of the whole accumulated batch, not per micro-batch.
1import torch, torch.nn.functional as F2from transformers import AutoTokenizer, AutoModelForCausalLM34MODEL = "Qwen/Qwen2.5-0.5B" # any base model whose tokenizer ships a chat template5tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")6model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16,7 attn_implementation="flash_attention_2").cuda()89def encode(messages, max_len=2048):10 """input_ids for the whole conversation; labels = -100 except assistant turns (+ their end token)."""11 ids = tok.apply_chat_template(messages, tokenize=True)12 labels = [-100] * len(ids)13 for i, m in enumerate(messages):14 if m["role"] != "assistant":15 continue16 start = len(tok.apply_chat_template(messages[:i], tokenize=True, add_generation_prompt=True))17 end = len(tok.apply_chat_template(messages[:i + 1], tokenize=True))18 labels[start:end] = ids[start:end] # response tokens and <|im_end|> are targets19 return ids[:max_len], labels[:max_len]2021def pack(examples):22 """Flatten a list of (ids, labels) into one row; position_ids restart per example.23 With flash_attention_2, transformers reads the resets as sequence boundaries (varlen attention)."""24 ids, labels, pos = [], [], []25 for x, y in examples:26 ids += x27 labels += [-100] + y[1:] # never predict across an example boundary28 pos += list(range(len(x)))29 t = lambda v: torch.tensor([v]).cuda()30 return {"input_ids": t(ids), "labels": t(labels), "position_ids": t(pos)}3132data = [33 [{"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4."}],34 [{"role": "system", "content": "Answer in French."},35 {"role": "user", "content": "Capital of Italy?"}, {"role": "assistant", "content": "Rome."}],36]37opt = torch.optim.AdamW(model.parameters(), lr=1e-5, betas=(0.9, 0.95), weight_decay=0.1)3839batch = pack([encode(c) for c in data])40logits = model(input_ids=batch["input_ids"], position_ids=batch["position_ids"]).logits41# shift: the prediction at position t is scored on token t+1; -100 positions are ignored42loss = F.cross_entropy(logits[:, :-1].flatten(0, 1).float(), batch["labels"][:, 1:].flatten(),43 ignore_index=-100, reduction="sum")44n_targets = (batch["labels"][:, 1:] != -100).sum()45(loss / n_targets).backward() # token mean over the batch46opt.step(); opt.zero_grad()4748# inference must use the same template, ending with the assistant header49prompt = tok.apply_chat_template([{"role": "user", "content": "What is 3+3?"}],50 add_generation_prompt=True, return_tensors="pt").cuda()51print(tok.decode(model.generate(prompt, max_new_tokens=16)[0, prompt.shape[1]:]))Use when
- +Simple, stable, ordinary cross-entropy training; no reward model or sampling
- +Small, high-quality data sets (1K–100K examples) are enough to change behaviour
- +Sets the chat format and stop behaviour that later RLHF or DPO stages build on
- +Packing with document masks makes it compute-efficient with no change to the result
Avoid when
- −Imitates the demonstrations, including their errors and style; it cannot exceed them
- −Only shows good answers, never contrasts them with bad ones: preference methods add that
- −Overfits small data quickly, and validation loss is a poor checkpoint signal
- −Narrow data causes forgetting of general skills; facts the base lacks encourage hallucination
Check yourself
You pack examples into rows of 4096 tokens with a plain causal mask and positions 0…4095. What goes wrong?
Check yourself
Which tokens should carry a training label in a standard assistant-only SFT setup?
Remember
- ✓SFT minimises over response tokens only. Prompt and template labels are , and the end-of-turn token is a target.
- ✓Chat templates (ChatML, Llama's [INST] or header tokens) must be identical in training and inference. Use apply_chat_template with add_generation_prompt=True at inference.
- ✓Packing removes padding waste, but needs a block-diagonal mask or reset position_ids plus a varlen kernel. Without one, most attention pairs cross examples.
- ✓Typical recipe: learning rate 1e-5–2e-5 (1e-4–2e-4 with LoRA) for 2–3 epochs. Validation loss overfits early, so choose checkpoints by generation quality.
- ✓Quality beats quantity. LIMA used 1,000 curated examples, and Llama 2 stopped at 27,540. SFT mostly teaches format, while knowledge comes from pretraining.