# Supervised fine-tuning (SFT)

> SFT trains a pretrained LM on chat-templated (prompt, response) demonstrations with next-token cross-entropy on the response tokens only (prompt labels −100), usually for 2–3 epochs at a learning rate near 1e-5–2e-5. LIMA got strong results from 1,000 curated examples.

Source: https://www.mlfoundry.in/learn/supervised-fine-tuning-sft  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Supervised fine-tuning (SFT):** is the stage that turns a pretrained next-token predictor into an assistant. You continue training on demonstrations: pairs of a prompt $x$ (the system message, user turns and any earlier context) and a target response $y$ written by a human or a stronger model. The loss is the usual causal language-modelling cross-entropy, computed **only on the response tokens**. The modern recipe comes from InstructGPT (Ouyang et al., "Training language models to follow instructions with human feedback", NeurIPS 2022), where SFT on about 13k labeller demonstrations was step 1, before reward modelling and **RLHF**. Llama 2-Chat (Touvron et al., 2023) and LIMA (Zhou et al., "LIMA: Less Is More for Alignment", NeurIPS 2023) are the reference points for data scale and quality.

A pretrained model continues documents. Ask GPT-3 "Explain the moon landing to a 6 year old" and it may continue with more questions like it, as if it were writing a list of prompts. It has the knowledge but not the behaviour. Few-shot prompting can coax out the behaviour, but it is brittle and uses up context. **Pretraining and continued pretraining** changes what the model knows. SFT changes the format of what it does: answer in the assistant role, follow the instruction, stop when done.

The name predates chat models. GPT-1 (Radford et al., 2018) already called its second stage "supervised fine-tuning", with a task head trained on labelled data. **Instruction tuning** (FLAN, T0, 2021–22) fine-tuned on many NLP tasks phrased as instructions, to get zero-shot transfer. InstructGPT fine-tuned on open-ended demonstrations of what users actually asked an API. The 1.3B InstructGPT model, after the full SFT + RLHF pipeline, was preferred by labellers over the 175B GPT-3. Today "SFT" usually means this chat-demonstration stage, run before preference tuning with **RLHF** or **Direct preference optimization (DPO)**.

## Objective and loss masking

$$
\mathcal L_{SFT}(\theta)=-\frac{1}{\sum_{i}|y^{(i)}|}\sum_{i}\sum_{t=1}^{|y^{(i)}|}\log p_\theta\big(y^{(i)}_t\,\big|\,x^{(i)},\,y^{(i)}_{<t}\big)
$$

- $x^{(i)}$: the prompt: system message, user turns, earlier assistant turns and template tokens. It is context, not a target
- $y^{(i)}$: the assistant response **including** its end-of-turn token, so the model learns when to stop
- $\sum_i|y^{(i)}|$: normalise by the number of target tokens in the batch (token mean), so long answers weigh more than short ones

In code, the mask is the label value $-100$, which PyTorch's `cross_entropy` ignores by default. Inputs are the full rendered conversation, and labels are a copy with every prompt and template position set to $-100$. The model shifts labels by one internally, so position $t$ is scored on token $t+1$. Llama 2-Chat describes it plainly: prompt and answer are concatenated with a special separator, and the loss on prompt tokens is zeroed out.

Why mask the prompt? User text is hard to predict. Learning to predict it spends capacity on imitating users, not on answering them. It also dominates the loss: in the worked example below, 5 prompt and template tokens contribute 92.7% of the unmasked loss sum. Masking is a default, not a law. Shi et al. ("Instruction Tuning With Loss Over Instructions", 2024) found that keeping the loss on instructions can help when responses are short and data is scarce.

## Chat templates

A conversation is a list of messages, `[{"role": "user", "content": ...}, {"role": "assistant", ...}]`. The model sees one token sequence. A **chat template** serialises roles with special tokens. ChatML (used by Qwen and others) writes `<|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n...<|im_end|>`. Llama 2 uses `[INST] <<SYS>>...<</SYS>> ... [/INST]`, and Llama 3 uses `<|start_header_id|>user<|end_header_id|>\n\n...<|eot_id|>`. In Hugging Face the template is a Jinja string stored with the tokenizer, rendered by `tokenizer.apply_chat_template`. With `add_generation_prompt=True` it appends the assistant header, so generation starts exactly where training targets started.

> Figure: The SFT data pipeline. Messages are rendered with a chat template and tokenised. Labels are −100 on everything except the assistant's content and its closing <|im_end|>, which must be trained or the model never learns to stop. Examples are then padded or packed into fixed-length rows. When packing, position_ids restart at 0 for each example, and attention must not cross example boundaries.

## Packing and attention masking

SFT examples vary wildly in length: a one-line answer, then a 2,000-token code review. Padding every example to the maximum length wastes most of the compute on pad tokens. **Packing** concatenates several examples into each row of length $S$. A causal mask alone then lets example $k$ attend to examples $1\ldots k-1$ in the same row. This is **cross-contamination**: the model learns to condition on unrelated text, and positional encodings no longer start at 0. Krell et al. ("Efficient Sequence Packing without Cross-contamination", 2021) showed that block-diagonal masks plus per-example position ids give packed training the same result as unpacked training.

$$
M_{ij}=\begin{cases}0 & j\le i\ \text{and}\ \mathrm{doc}(j)=\mathrm{doc}(i)\\-\infty&\text{otherwise}\end{cases},\qquad \text{pos}_i=i-\mathrm{start}(\mathrm{doc}(i))
$$

- $M$: additive attention mask for one packed row: causal **and** block-diagonal
- $\text{pos}_i$: position id restarts at each example, so RoPE sees the same positions as in an unpacked batch
- $\text{varlen}$: in practice nobody builds $M$: FlashAttention's variable-length kernels take cumulative sequence lengths and skip the off-diagonal blocks entirely (see **FlashAttention**)

Also set the label of each example's first token to $-100$. Otherwise the last token of one example is trained to predict the first token of the next. Hugging Face's `DataCollatorWithFlattening` does both: it concatenates examples, resets `position_ids`, and inserts a $-100$ separator label.

> Figure: Eight chat-templated conversations (321 tokens: system violet, user amber, assistant blue, the assistant's closing <|im_end|> red, template grey). Bright tokens are loss targets, and faded tokens carry label −100. At the default (pack into S = 128, assistant-only loss, no document mask), best-fit-decreasing packing needs 3 rows: 384 slots, 63 pad, utilisation 83.6%. 134 of the 321 real tokens (41.7%) are targets. Causal attention over real tokens allows 17,755 query–key pairs, and 10,609 of them (59.8%) cross into another conversation, shown in red in the row's mask. Turning on the document mask removes all of them, leaving 7,146 pairs. That is also all a varlen kernel computes, against 24,768 for dense masks over the full rows. Padding to the longest conversation (54) needs 8 rows, 432 slots and 111 pad (74.3%). Padding to S = 128 wastes 703 of 1,024 slots (31.3% used). Packing into S = 64 gives 6 rows at 83.6%, with 1,675 cross pairs. With assistant-only loss, the token mean gives each conversation between 5.2% and 23.1% of the gradient. With loss on every token, 321 targets are trained and the spread narrows to 7.2–16.8%.

## Hyperparameters

SFT is a small nudge on a large model. Learning rates are roughly an order of magnitude below pretraining peaks: $1\cdot10^{-5}$ to $2\cdot10^{-5}$ for full fine-tuning of 7B–70B models, with warmup and cosine or linear decay. **LoRA** runs typically use $1\cdot10^{-4}$ to $2\cdot10^{-4}$ because only the small adapters move. Two or three epochs are typical. More epochs drive training loss toward zero on a few thousand examples, and the model starts reciting them.

**Published SFT recipes**

- **InstructGPT (2022):** GPT-3 base models, about 13k training prompts with labeller demonstrations; 16 epochs, cosine decay, residual dropout 0.2. SFT models overfit on validation loss after 1 epoch, but more epochs still improved reward-model score and human preference
- **Alpaca (2023):** LLaMA 7B, 52K demonstrations generated by text-davinci-003; 3 epochs, learning rate $2\cdot10^{-5}$, batch 128, max length 512
- **LIMA (2023):** LLaMA 65B, **1,000** curated examples (750 from Stack Exchange, wikiHow and Reddit, 250 written by the authors); 15 epochs, AdamW $\beta_2=0.95$, weight decay 0.1, lr $10^{-5}$ decaying linearly to $10^{-6}$, batch 32, max 2048 tokens, residual dropout rising from 0.0 to 0.3 across layers
- **Llama 2-Chat (2023):** 27,540 vendor-written SFT annotations; cosine schedule, lr $2\cdot10^{-5}$, weight decay 0.1, batch 64, sequence length 4096, 2 epochs; prompt loss zeroed

## Data quality over quantity

LIMA fine-tuned LLaMA 65B on 1,000 carefully chosen examples, with no RLHF. In a human study, its answers were equivalent to or preferred over GPT-4's in 43% of cases. The authors proposed the **Superficial Alignment Hypothesis**: almost all knowledge is learned in pretraining, and SFT mainly teaches the format and style in which to present it. Llama 2's authors reached a similar conclusion. Setting aside millions of third-party examples in favour of fewer, higher-quality vendor annotations improved results notably, and they stopped at 27,540.

Quality here means correct, helpful answers in a consistent voice, diverse prompts, and no near-duplicates. LIMA's ablations found that prompt diversity and response quality mattered more than doubling quantity. A single bad pattern repeated across thousands of examples, such as a canned refusal or a stock opening, gets copied faithfully by the model.

## Overfitting and forgetting

Validation loss on held-out demonstrations is a weak guide. InstructGPT and LIMA both trained well past the point where it started rising, and chose checkpoints by human or reward-model judgement instead. LIMA selected checkpoints between epochs 5 and 10 on a 50-example development set. Generation quality, not perplexity, is the target.

**Catastrophic forgetting** is the other risk. A narrow SFT set, such as only code or only one language, can erode general abilities that the base model had. The mitigations are a low learning rate, few epochs, a broad data mixture (often with some general chat or pretraining-style data mixed in), or a parameter-efficient method. Biderman et al. ("LoRA Learns Less and Forgets Less", 2024) measured exactly that trade-off: LoRA learned a new domain less fully than full fine-tuning, but kept more of the base model's other skills. See **Parameter-efficient fine-tuning (PEFT)**.

## Worked example: what the mask changes

One short exchange, "What is 2+2?" → "4.", with `[A]` the assistant header and `[E]` the end-of-turn token. Each row is the probability the model assigns to that token as the next one.

Per-token negative log-likelihood (probabilities made up, NLL computed in Python)

| target token | p(token \| prefix) | −log p | label |
| --- | --- | --- | --- |
| What | 0.02 | 3.912 | −100 |
| is | 0.30 | 1.204 | −100 |
| 2+2 | 0.01 | 4.605 | −100 |
| ? | 0.40 | 0.916 | −100 |
| [A] | 0.95 | 0.051 | −100 |
| 4 | 0.90 | 0.105 | trained |
| . | 0.60 | 0.511 | trained |
| [E] | 0.80 | 0.223 | trained |

**Computing the loss**

1. Masked SFT loss: $(0.105+0.511+0.223)/3=0.839/3=0.280$.
2. Unmasked loss over all 8 targets: $11.528/8=1.441$. The prompt tokens supply $92.7\%$ of the sum, so the gradient would mostly teach the model to predict user questions.
3. Batch it with a second example that has 30 response tokens, each with NLL $0.2$. The token mean is $(0.839+30\cdot0.2)/33=0.207$, and the short example gets $3/33=9.1\%$ of the weight.
4. A per-example mean, $(0.280+0.200)/2=0.240$, would give each example half. Both are used. Pick one deliberately, and with gradient accumulation normalise by the target count of the whole accumulated batch, not per micro-batch.

```python
import torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "Qwen/Qwen2.5-0.5B"                      # any base model whose tokenizer ships a chat template
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16,
                                             attn_implementation="flash_attention_2").cuda()

def encode(messages, max_len=2048):
    """input_ids for the whole conversation; labels = -100 except assistant turns (+ their end token)."""
    ids = tok.apply_chat_template(messages, tokenize=True)
    labels = [-100] * len(ids)
    for i, m in enumerate(messages):
        if m["role"] != "assistant":
            continue
        start = len(tok.apply_chat_template(messages[:i], tokenize=True, add_generation_prompt=True))
        end = len(tok.apply_chat_template(messages[:i + 1], tokenize=True))
        labels[start:end] = ids[start:end]            # response tokens and <|im_end|> are targets
    return ids[:max_len], labels[:max_len]

def pack(examples):
    """Flatten a list of (ids, labels) into one row; position_ids restart per example.
    With flash_attention_2, transformers reads the resets as sequence boundaries (varlen attention)."""
    ids, labels, pos = [], [], []
    for x, y in examples:
        ids += x
        labels += [-100] + y[1:]                      # never predict across an example boundary
        pos += list(range(len(x)))
    t = lambda v: torch.tensor([v]).cuda()
    return {"input_ids": t(ids), "labels": t(labels), "position_ids": t(pos)}

data = [
    [{"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4."}],
    [{"role": "system", "content": "Answer in French."},
     {"role": "user", "content": "Capital of Italy?"}, {"role": "assistant", "content": "Rome."}],
]
opt = torch.optim.AdamW(model.parameters(), lr=1e-5, betas=(0.9, 0.95), weight_decay=0.1)

batch = pack([encode(c) for c in data])
logits = model(input_ids=batch["input_ids"], position_ids=batch["position_ids"]).logits
# shift: the prediction at position t is scored on token t+1; -100 positions are ignored
loss = F.cross_entropy(logits[:, :-1].flatten(0, 1).float(), batch["labels"][:, 1:].flatten(),
                       ignore_index=-100, reduction="sum")
n_targets = (batch["labels"][:, 1:] != -100).sum()
(loss / n_targets).backward()                         # token mean over the batch
opt.step(); opt.zero_grad()

# inference must use the same template, ending with the assistant header
prompt = tok.apply_chat_template([{"role": "user", "content": "What is 3+3?"}],
                                 add_generation_prompt=True, return_tensors="pt").cuda()
print(tok.decode(model.generate(prompt, max_new_tokens=16)[0, prompt.shape[1]:]))
```

SFT with Hugging Face Transformers: assistant-only labels from a chat template, packing with reset position_ids, and the masked loss (not run here)

> **Train the end-of-turn token:** If the mask also hides `<|im_end|>` (or `</s>`, `<|eot_id|>`), the model never sees a target that says "stop here". It then rambles into a fake next user turn. The opposite bug is a template mismatch: training with one template and serving with another, or adding BOS twice. Both silently degrade quality. Always decode one rendered training example and read it.

> **SFT teaches format, pretraining supplies the knowledge:** LIMA's 1,000 examples of at most 2,048 tokens are about $10^{-6}$ of LLaMA 65B's 1.4T pretraining tokens, yet they change the model's behaviour completely. That is only possible if the skills already exist and SFT selects how to present them. It also explains the failure mode: SFT on facts the base model does not know teaches it to state confident answers it cannot ground, which encourages hallucination.

> **Packing changes the loss weighting:** With padding and a per-row mean, each example weighs the same. With packing and a token mean, weight follows response length, and in the figure one conversation gets 4.4 times another's share. Neither is wrong, but switching between them changes what the model optimises. Report which one you used.

**Strengths**

- Simple, stable, ordinary cross-entropy training; no reward model or sampling
- Small, high-quality data sets (1K–100K examples) are enough to change behaviour
- Sets the chat format and stop behaviour that later RLHF or DPO stages build on
- Packing with document masks makes it compute-efficient with no change to the result

**Limitations**

- Imitates the demonstrations, including their errors and style; it cannot exceed them
- Only shows good answers, never contrasts them with bad ones: preference methods add that
- Overfits small data quickly, and validation loss is a poor checkpoint signal
- Narrow data causes forgetting of general skills; facts the base lacks encourage hallucination

**Question:** You pack examples into rows of 4096 tokens with a plain causal mask and positions 0…4095. What goes wrong?

- A. Nothing; the causal mask already separates examples
- B. Tokens attend to earlier, unrelated examples in the row, and positions no longer start at 0 for each example
- C. The loss mask is ignored
- D. The model cannot learn the end-of-turn token

**Answer:** B. Tokens attend to earlier, unrelated examples in the row, and positions no longer start at 0 for each example — A causal mask only blocks the future. Without a block-diagonal document mask, or `position_ids` that reset per example with a varlen kernel, example $k$ conditions on examples $1\ldots k-1$. In the figure at S = 128 that is 59.8% of all attention pairs. Its RoPE positions also start wherever it landed in the row.

**Question:** Which tokens should carry a training label in a standard assistant-only SFT setup?

- A. All tokens, including the user prompt
- B. Only the user prompt
- C. The assistant's response tokens and its end-of-turn token
- D. Only the final token of the response

**Answer:** C. The assistant's response tokens and its end-of-turn token — The prompt is context, so its labels are −100. The response is the target, and its end-of-turn token must be included so the model learns to stop. In the worked example, the masked loss is 0.280 against 1.441 unmasked, where 92.7% of the unmasked sum came from predicting the user's words.

**Key points**

- SFT minimises $-\sum_t\log p_\theta(y_t\mid x,y_{<t})$ over response tokens only. Prompt and template labels are $-100$, and the end-of-turn token is a target.
- Chat templates (ChatML, Llama's [INST] or header tokens) must be identical in training and inference. Use apply_chat_template with add_generation_prompt=True at inference.
- Packing removes padding waste, but needs a block-diagonal mask or reset position_ids plus a varlen kernel. Without one, most attention pairs cross examples.
- Typical recipe: learning rate 1e-5–2e-5 (1e-4–2e-4 with LoRA) for 2–3 epochs. Validation loss overfits early, so choose checkpoints by generation quality.
- Quality beats quantity. LIMA used 1,000 curated examples, and Llama 2 stopped at 27,540. SFT mostly teaches format, while knowledge comes from pretraining.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
