Supervised fine-tuning (SFT)

SFT trains a pretrained LM on chat-templated (prompt, response) demonstrations with next-token cross-entropy on the response tokens only (prompt labels −100), usually for 2–3 epochs at a learning rate near 1e-5–2e-5. LIMA got strong results from 1,000 curated examples.

15 min read1436 wordscore0 viewsupdated 2026-10-13

Definition

Supervised fine-tuning (SFT) is the stage that turns a pretrained next-token predictor into an assistant. You continue training on demonstrations: pairs of a prompt xx (the system message, user turns and any earlier context) and a target response yy written by a human or a stronger model. The loss is the usual causal language-modelling cross-entropy, computed only on the response tokens. The modern recipe comes from InstructGPT (Ouyang et al., "Training language models to follow instructions with human feedback", NeurIPS 2022), where SFT on about 13k labeller demonstrations was step 1, before reward modelling and RLHF. Llama 2-Chat (Touvron et al., 2023) and LIMA (Zhou et al., "LIMA: Less Is More for Alignment", NeurIPS 2023) are the reference points for data scale and quality.

A pretrained model continues documents. Ask GPT-3 "Explain the moon landing to a 6 year old" and it may continue with more questions like it, as if it were writing a list of prompts. It has the knowledge but not the behaviour. Few-shot prompting can coax out the behaviour, but it is brittle and uses up context. Pretraining and continued pretraining changes what the model knows. SFT changes the format of what it does: answer in the assistant role, follow the instruction, stop when done.

The name predates chat models. GPT-1 (Radford et al., 2018) already called its second stage "supervised fine-tuning", with a task head trained on labelled data. Instruction tuning (FLAN, T0, 2021–22) fine-tuned on many NLP tasks phrased as instructions, to get zero-shot transfer. InstructGPT fine-tuned on open-ended demonstrations of what users actually asked an API. The 1.3B InstructGPT model, after the full SFT + RLHF pipeline, was preferred by labellers over the 175B GPT-3. Today "SFT" usually means this chat-demonstration stage, run before preference tuning with RLHF or Direct preference optimization (DPO).

Objective and loss masking

LSFT(θ)=−1∑i∣y(i)∣∑i∑t=1∣y(i)∣log⁡pθ(yt(i) ∣ x(i), y<t(i))\mathcal L_{SFT}(\theta)=-\frac{1}{\sum_{i}|y^{(i)}|}\sum_{i}\sum_{t=1}^{|y^{(i)}|}\log p_\theta\big(y^{(i)}_t\,\big|\,x^{(i)},\,y^{(i)}_{<t}\big)
x(i)x^{(i)}
the prompt: system message, user turns, earlier assistant turns and template tokens. It is context, not a target
y(i)y^{(i)}
the assistant response including its end-of-turn token, so the model learns when to stop
∑i∣y(i)∣\sum_i|y^{(i)}|
normalise by the number of target tokens in the batch (token mean), so long answers weigh more than short ones

In code, the mask is the label value −100-100, which PyTorch's cross_entropy ignores by default. Inputs are the full rendered conversation, and labels are a copy with every prompt and template position set to −100-100. The model shifts labels by one internally, so position tt is scored on token t+1t+1. Llama 2-Chat describes it plainly: prompt and answer are concatenated with a special separator, and the loss on prompt tokens is zeroed out.

Why mask the prompt? User text is hard to predict. Learning to predict it spends capacity on imitating users, not on answering them. It also dominates the loss: in the worked example below, 5 prompt and template tokens contribute 92.7% of the unmasked loss sum. Masking is a default, not a law. Shi et al. ("Instruction Tuning With Loss Over Instructions", 2024) found that keeping the loss on instructions can help when responses are short and data is scarce.

Chat templates

A conversation is a list of messages, [{"role": "user", "content": ...}, {"role": "assistant", ...}]. The model sees one token sequence. A chat template serialises roles with special tokens. ChatML (used by Qwen and others) writes <|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n...<|im_end|>. Llama 2 uses [INST] <<SYS>>...<</SYS>> ... [/INST], and Llama 3 uses <|start_header_id|>user<|end_header_id|>\n\n...<|eot_id|>. In Hugging Face the template is a Jinja string stored with the tokenizer, rendered by tokenizer.apply_chat_template. With add_generation_prompt=True it appends the assistant header, so generation starts exactly where training targets started.

messagesrole, contentchat templateChatML, [INST]tokenizeids ≤ max_lenloss masklabels = −100pack or padposition_idscausal LMCE on targetsone rendered example (illustrative token split)<|im_start|>−100user−100\n−100What−100is−100the−100capital−100of−100France−100?−100<|im_end|>−100\n−100<|im_start|>−100assistant−100\n−100ThelosscapitallossislossParisloss.loss<|im_end|>loss\n−100prompt: contextanswer + stop tokenL = −(1/|A|) Σ_(t∈A) log p_θ(x_t | x_<t), A = assistant tokens and their <|im_end|>batching three examples (lengths 9, 5, 4) into rows of S = 10padded3 rows, 30 slots, 12 pad (60% used)packed012345678012340123digits = position_ids, reset per example2 rows, 20 slots, 2 pad (90% used)Without a document mask (or position_ids reset with a varlen attention kernel), theviolet example in row 2 can attend to the mint one before it: cross-contamination.
fig.The SFT data pipeline. Messages are rendered with a chat template and tokenised. Labels are −100 on everything except the assistant's content and its closing <|im_end|>, which must be trained or the model never learns to stop. Examples are then padded or packed into fixed-length rows. When packing, position_ids restart at 0 for each example, and attention must not cross example boundaries.

Packing and attention masking

SFT examples vary wildly in length: a one-line answer, then a 2,000-token code review. Padding every example to the maximum length wastes most of the compute on pad tokens. Packing concatenates several examples into each row of length SS. A causal mask alone then lets example kk attend to examples 1…k−11\ldots k-1 in the same row. This is cross-contamination: the model learns to condition on unrelated text, and positional encodings no longer start at 0. Krell et al. ("Efficient Sequence Packing without Cross-contamination", 2021) showed that block-diagonal masks plus per-example position ids give packed training the same result as unpacked training.

Mij={0j≤i and doc(j)=doc(i)−∞otherwise,posi=i−start(doc(i))M_{ij}=\begin{cases}0 & j\le i\ \text{and}\ \mathrm{doc}(j)=\mathrm{doc}(i)\\-\infty&\text{otherwise}\end{cases},\qquad \text{pos}_i=i-\mathrm{start}(\mathrm{doc}(i))
MM
additive attention mask for one packed row: causal and block-diagonal
posi\text{pos}_i
position id restarts at each example, so RoPE sees the same positions as in an unpacked batch
varlen\text{varlen}
in practice nobody builds MM: FlashAttention's variable-length kernels take cumulative sequence lengths and skip the off-diagonal blocks entirely (see FlashAttention)

Also set the label of each example's first token to −100-100. Otherwise the last token of one example is trained to predict the first token of the next. Hugging Face's DataCollatorWithFlattening does both: it concatenates examples, resets position_ids, and inserts a −100-100 separator label.

8 conversations · 3 rows × 128 tokens · click a row for its attention mask■ system ■ user ■ assistant ■ <|im_end|> after assistant ■ template · bright = in loss, faded = label −100row 0 attention (query ↓, key →)■ same conversation ■ leaks acrossslots 3 × 128 = 384real tokens 321 · pad 63 · utilisation 83.6%loss targets 134 of 321 real (41.7%)system, user and template tokens carry label −100causal pairs over real tokens 17,755cross-conversation pairs 10,609 (59.8%)attention work ∝ 24,768 dense · 7,146 varlentoken-mean loss weight per conversation 5.2% to 23.1%
batching
masks
fig.Eight chat-templated conversations (321 tokens: system violet, user amber, assistant blue, the assistant's closing <|im_end|> red, template grey). Bright tokens are loss targets, and faded tokens carry label −100. At the default (pack into S = 128, assistant-only loss, no document mask), best-fit-decreasing packing needs 3 rows: 384 slots, 63 pad, utilisation 83.6%. 134 of the 321 real tokens (41.7%) are targets. Causal attention over real tokens allows 17,755 query–key pairs, and 10,609 of them (59.8%) cross into another conversation, shown in red in the row's mask. Turning on the document mask removes all of them, leaving 7,146 pairs. That is also all a varlen kernel computes, against 24,768 for dense masks over the full rows. Padding to the longest conversation (54) needs 8 rows, 432 slots and 111 pad (74.3%). Padding to S = 128 wastes 703 of 1,024 slots (31.3% used). Packing into S = 64 gives 6 rows at 83.6%, with 1,675 cross pairs. With assistant-only loss, the token mean gives each conversation between 5.2% and 23.1% of the gradient. With loss on every token, 321 targets are trained and the spread narrows to 7.2–16.8%.

Hyperparameters

SFT is a small nudge on a large model. Learning rates are roughly an order of magnitude below pretraining peaks: 1⋅10−51\cdot10^{-5} to 2⋅10−52\cdot10^{-5} for full fine-tuning of 7B–70B models, with warmup and cosine or linear decay. LoRA runs typically use 1⋅10−41\cdot10^{-4} to 2⋅10−42\cdot10^{-4} because only the small adapters move. Two or three epochs are typical. More epochs drive training loss toward zero on a few thousand examples, and the model starts reciting them.

Published SFT recipes

InstructGPT (2022)
GPT-3 base models, about 13k training prompts with labeller demonstrations; 16 epochs, cosine decay, residual dropout 0.2. SFT models overfit on validation loss after 1 epoch, but more epochs still improved reward-model score and human preference
Alpaca (2023)
LLaMA 7B, 52K demonstrations generated by text-davinci-003; 3 epochs, learning rate 2⋅10−52\cdot10^{-5}, batch 128, max length 512
LIMA (2023)
LLaMA 65B, 1,000 curated examples (750 from Stack Exchange, wikiHow and Reddit, 250 written by the authors); 15 epochs, AdamW β2=0.95\beta_2=0.95, weight decay 0.1, lr 10−510^{-5} decaying linearly to 10−610^{-6}, batch 32, max 2048 tokens, residual dropout rising from 0.0 to 0.3 across layers
Llama 2-Chat (2023)
27,540 vendor-written SFT annotations; cosine schedule, lr 2⋅10−52\cdot10^{-5}, weight decay 0.1, batch 64, sequence length 4096, 2 epochs; prompt loss zeroed

Data quality over quantity

LIMA fine-tuned LLaMA 65B on 1,000 carefully chosen examples, with no RLHF. In a human study, its answers were equivalent to or preferred over GPT-4's in 43% of cases. The authors proposed the Superficial Alignment Hypothesis: almost all knowledge is learned in pretraining, and SFT mainly teaches the format and style in which to present it. Llama 2's authors reached a similar conclusion. Setting aside millions of third-party examples in favour of fewer, higher-quality vendor annotations improved results notably, and they stopped at 27,540.

Quality here means correct, helpful answers in a consistent voice, diverse prompts, and no near-duplicates. LIMA's ablations found that prompt diversity and response quality mattered more than doubling quantity. A single bad pattern repeated across thousands of examples, such as a canned refusal or a stock opening, gets copied faithfully by the model.

Overfitting and forgetting

Validation loss on held-out demonstrations is a weak guide. InstructGPT and LIMA both trained well past the point where it started rising, and chose checkpoints by human or reward-model judgement instead. LIMA selected checkpoints between epochs 5 and 10 on a 50-example development set. Generation quality, not perplexity, is the target.

Catastrophic forgetting is the other risk. A narrow SFT set, such as only code or only one language, can erode general abilities that the base model had. The mitigations are a low learning rate, few epochs, a broad data mixture (often with some general chat or pretraining-style data mixed in), or a parameter-efficient method. Biderman et al. ("LoRA Learns Less and Forgets Less", 2024) measured exactly that trade-off: LoRA learned a new domain less fully than full fine-tuning, but kept more of the base model's other skills. See Parameter-efficient fine-tuning (PEFT).

Worked example: what the mask changes

One short exchange, "What is 2+2?" → "4.", with [A] the assistant header and [E] the end-of-turn token. Each row is the probability the model assigns to that token as the next one.

target tokenp(token | prefix)−log plabel
What0.023.912−100
is0.301.204−100
2+20.014.605−100
?0.400.916−100
[A]0.950.051−100
40.900.105trained
.0.600.511trained
[E]0.800.223trained
Per-token negative log-likelihood (probabilities made up, NLL computed in Python)

Computing the loss

  1. 1Masked SFT loss: (0.105+0.511+0.223)/3=0.839/3=0.280(0.105+0.511+0.223)/3=0.839/3=0.280.
  2. 2Unmasked loss over all 8 targets: 11.528/8=1.44111.528/8=1.441. The prompt tokens supply 92.7%92.7\% of the sum, so the gradient would mostly teach the model to predict user questions.
  3. 3Batch it with a second example that has 30 response tokens, each with NLL 0.20.2. The token mean is (0.839+30⋅0.2)/33=0.207(0.839+30\cdot0.2)/33=0.207, and the short example gets 3/33=9.1%3/33=9.1\% of the weight.
  4. 4A per-example mean, (0.280+0.200)/2=0.240(0.280+0.200)/2=0.240, would give each example half. Both are used. Pick one deliberately, and with gradient accumulation normalise by the target count of the whole accumulated batch, not per micro-batch.
python
1import torch, torch.nn.functional as F
2from transformers import AutoTokenizer, AutoModelForCausalLM
3
4MODEL = "Qwen/Qwen2.5-0.5B" # any base model whose tokenizer ships a chat template
5tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
6model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16,
7 attn_implementation="flash_attention_2").cuda()
8
9def encode(messages, max_len=2048):
10 """input_ids for the whole conversation; labels = -100 except assistant turns (+ their end token)."""
11 ids = tok.apply_chat_template(messages, tokenize=True)
12 labels = [-100] * len(ids)
13 for i, m in enumerate(messages):
14 if m["role"] != "assistant":
15 continue
16 start = len(tok.apply_chat_template(messages[:i], tokenize=True, add_generation_prompt=True))
17 end = len(tok.apply_chat_template(messages[:i + 1], tokenize=True))
18 labels[start:end] = ids[start:end] # response tokens and <|im_end|> are targets
19 return ids[:max_len], labels[:max_len]
20
21def pack(examples):
22 """Flatten a list of (ids, labels) into one row; position_ids restart per example.
23 With flash_attention_2, transformers reads the resets as sequence boundaries (varlen attention)."""
24 ids, labels, pos = [], [], []
25 for x, y in examples:
26 ids += x
27 labels += [-100] + y[1:] # never predict across an example boundary
28 pos += list(range(len(x)))
29 t = lambda v: torch.tensor([v]).cuda()
30 return {"input_ids": t(ids), "labels": t(labels), "position_ids": t(pos)}
31
32data = [
33 [{"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4."}],
34 [{"role": "system", "content": "Answer in French."},
35 {"role": "user", "content": "Capital of Italy?"}, {"role": "assistant", "content": "Rome."}],
36]
37opt = torch.optim.AdamW(model.parameters(), lr=1e-5, betas=(0.9, 0.95), weight_decay=0.1)
38
39batch = pack([encode(c) for c in data])
40logits = model(input_ids=batch["input_ids"], position_ids=batch["position_ids"]).logits
41# shift: the prediction at position t is scored on token t+1; -100 positions are ignored
42loss = F.cross_entropy(logits[:, :-1].flatten(0, 1).float(), batch["labels"][:, 1:].flatten(),
43 ignore_index=-100, reduction="sum")
44n_targets = (batch["labels"][:, 1:] != -100).sum()
45(loss / n_targets).backward() # token mean over the batch
46opt.step(); opt.zero_grad()
47
48# inference must use the same template, ending with the assistant header
49prompt = tok.apply_chat_template([{"role": "user", "content": "What is 3+3?"}],
50 add_generation_prompt=True, return_tensors="pt").cuda()
51print(tok.decode(model.generate(prompt, max_new_tokens=16)[0, prompt.shape[1]:]))
SFT with Hugging Face Transformers: assistant-only labels from a chat template, packing with reset position_ids, and the masked loss (not run here)

Use when

  • +Simple, stable, ordinary cross-entropy training; no reward model or sampling
  • +Small, high-quality data sets (1K–100K examples) are enough to change behaviour
  • +Sets the chat format and stop behaviour that later RLHF or DPO stages build on
  • +Packing with document masks makes it compute-efficient with no change to the result

Avoid when

  • −Imitates the demonstrations, including their errors and style; it cannot exceed them
  • −Only shows good answers, never contrasts them with bad ones: preference methods add that
  • −Overfits small data quickly, and validation loss is a poor checkpoint signal
  • −Narrow data causes forgetting of general skills; facts the base lacks encourage hallucination

Check yourself

You pack examples into rows of 4096 tokens with a plain causal mask and positions 0…4095. What goes wrong?

Check yourself

Which tokens should carry a training label in a standard assistant-only SFT setup?

Remember

  • ✓SFT minimises −∑tlog⁡pθ(yt∣x,y<t)-\sum_t\log p_\theta(y_t\mid x,y_{<t}) over response tokens only. Prompt and template labels are −100-100, and the end-of-turn token is a target.
  • ✓Chat templates (ChatML, Llama's [INST] or header tokens) must be identical in training and inference. Use apply_chat_template with add_generation_prompt=True at inference.
  • ✓Packing removes padding waste, but needs a block-diagonal mask or reset position_ids plus a varlen kernel. Without one, most attention pairs cross examples.
  • ✓Typical recipe: learning rate 1e-5–2e-5 (1e-4–2e-4 with LoRA) for 2–3 epochs. Validation loss overfits early, so choose checkpoints by generation quality.
  • ✓Quality beats quantity. LIMA used 1,000 curated examples, and Llama 2 stopped at 27,540. SFT mostly teaches format, while knowledge comes from pretraining.
← back to Deep Learning roadmap