Definition
Direct preference optimization (DPO) is the preference-tuning method of Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (NeurIPS 2023). It trains a policy directly on preference pairs with a binary cross-entropy loss on the difference of log-probability ratios against a frozen reference . It has no reward model, no sampling during training, and no RL. At its optimum it gives the same policy as RLHF with a KL penalty, under the Bradley–Terry preference model.
RLHF works, but it is heavy. You train a reward model, then run Proximal policy optimization with four networks in memory: the policy, the reference, the reward model and the value model. You also have to generate text inside the training loop. PPO for language models is notoriously sensitive to hyperparameters. Rafailov et al. noticed that the KL-regularised objective RLHF optimises has a closed-form solution. That solution can be inverted to express the reward in terms of the policy. Substituting it into the reward model's own loss gives a loss on the policy alone: the reward-modelling step is the policy-optimisation step.
Derivation
- the (unknown) reward behind human preferences
- the reference policy, normally the Supervised fine-tuning (SFT) model
- strength of the KL constraint, the same role as in RLHF
Write the objective for one prompt as a single expectation and divide by . Maximising it is the same as minimising . Define and . The objective becomes . does not depend on , and a KL is zero only when the two distributions are equal, so:
- a sum over every possible response, which is intractable. This is why RLHF approximates with PPO instead of computing it
- depends only on the prompt, so it cancels in any comparison of two responses to the same prompt
Now use the Bradley–Terry model, , the same one used to train RLHF reward models. Only reward differences appear, so drops out. Replace the unknown with the trainable , and maximise the likelihood of the observed preferences:
- the implicit reward: the policy is its own reward model
- the sum of token log-probabilities of the response given the prompt; one forward pass per model per response
- the preferred ("won") and dispreferred ("lost") responses
The paper also proves that nothing is lost by this reparameterisation. Every reward function is equivalent, under Bradley–Terry, to one of the form for some policy : two rewards that differ by a function of alone induce the same preferences and the same optimal policy. Fitting a reward model and then solving the KL-constrained RL problem therefore has the same population optimum as minimising directly.
What the gradient does
- 0.5 at the start (where ), near 1 if the implicit reward ranks the pair the wrong way, near 0 once the margin is large
- scales both the margin inside and the overall gradient
The weighting is what separates DPO from a naive "push up the good, push down the bad" (unlikelihood) objective. Without it, every pair gets pushed forever, and the paper reports that the model degenerates. With it, pairs that are already ranked correctly by a wide margin stop contributing. Learning concentrates on pairs where the implicit reward disagrees with the human label. The weight is the gradient of logistic regression, which is exactly what DPO is: logistic regression of the preference labels on the implicit reward margin.
The role of β
appears in two places. It sets the target: the optimum is , so a small allows a large departure from the reference and a large keeps the policy close. It also sets the scale: the log-ratio margin needed to reach a given loss is proportional to . With , a margin of nats is only in implicit-reward units, which still leaves loss . The paper used in most experiments and for TL;DR summarisation.
DPO in the paper and in practice
- Tasks
- controlled sentiment generation on IMDb (GPT-2-large, with a sentiment classifier as the true reward), TL;DR summarisation, and single-turn dialogue on Anthropic HH (Pythia-2.8B)
- Hyperparameters
- (0.5 for TL;DR), batch size 64, RMSprop at learning rate with 150 warm-up steps
- Reference
- the SFT model. When none existed (HH), they fine-tuned the base model on the preferred responses first ("Preferred-FT")
- Results
- on IMDb, DPO's reward-vs-KL frontier dominated PPO's, including PPO with access to the true reward. On TL;DR, GPT-4-judged win rate against reference summaries was about 61% for DPO vs 57% for PPO at temperature 0, and DPO was more robust to sampling temperature
- Cost per pair
- 4 forward passes ( and on and ), backward through only; log-probs can be precomputed once
- Adoption
- Zephyr-7B (DPO on UltraFeedback), Tülu 2, and Llama 3's post-training rounds (SFT, rejection sampling, DPO)
Worked example: one pair
Take and one pair whose summed log-probabilities are and , and .
Loss and gradient weight
- 1Implicit rewards: , .
- 2Margin . Loss , slightly below the starting value .
- 3Gradient weight , so the update is times .
- 4Note that has lost 0.8 nats relative to the reference while gained only 0.5. DPO only sees the difference: it would be equally happy if both went down, as long as went down more.
| m = r̂_w − r̂_l | −log σ(m) | weight σ(−m) |
|---|---|---|
| −2 | 2.1269 | 0.8808 |
| −1 | 1.3133 | 0.7311 |
| 0 (start, π_θ = π_ref) | 0.6931 | 0.5000 |
| 1 | 0.3133 | 0.2689 |
| 2 | 0.1269 | 0.1192 |
| 3 | 0.0486 | 0.0474 |
| 5 | 0.0067 | 0.0067 |
1import torch, torch.nn.functional as F23def seq_logprob(model, input_ids, attention_mask, labels):4 """Sum of log p(token) over response tokens. labels = input_ids with prompt and padding set to -100."""5 logits = model(input_ids=input_ids, attention_mask=attention_mask).logits[:, :-1]6 tgt = labels[:, 1:]7 mask = tgt != -1008 logp = torch.gather(logits.log_softmax(-1), 2, tgt.clamp(min=0).unsqueeze(-1)).squeeze(-1)9 return (logp * mask).sum(-1) # (B,)1011def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1):12 """All inputs: (B,) summed log-probs of the chosen (w) and rejected (l) responses."""13 r_w = beta * (pi_w - ref_w) # implicit rewards14 r_l = beta * (pi_l - ref_l)15 loss = -F.logsigmoid(r_w - r_l).mean()16 return loss, r_w.detach(), r_l.detach()1718loss, r_w, r_l = dpo_loss(torch.tensor([-12.0]), torch.tensor([-14.0]),19 torch.tensor([-12.5]), torch.tensor([-13.2]))20print(loss, r_w, r_l) # loss 0.6303, r_w 0.05, r_l -0.08 (the worked example)2122# training step (policy trainable, reference frozen, both start as the SFT model)23# with torch.no_grad():24# ref_w = seq_logprob(ref, ids_w, mask_w, labels_w); ref_l = seq_logprob(ref, ids_l, mask_l, labels_l)25# pi_w = seq_logprob(policy, ids_w, mask_w, labels_w); pi_l = seq_logprob(policy, ids_l, mask_l, labels_l)26# loss, r_w, r_l = dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1)27# loss.backward(); opt.step(); opt.zero_grad()28# log: reward accuracy (r_w > r_l).float().mean(), margin (r_w - r_l).mean(), and pi_w itself2930def ipo_loss(pi_w, pi_l, ref_w, ref_l, tau=0.1):31 h = (pi_w - ref_w) - (pi_l - ref_l) # log-ratio margin32 return ((h - 1 / (2 * tau)) ** 2).mean() # regress to a fixed margin3334# Libraries: trl.DPOTrainer implements this loss (and IPO/KTO variants) on (prompt, chosen, rejected) data.Weaknesses, and IPO and KTO
Deterministic preferences break the KL constraint. If always beats , the Bradley–Terry likelihood is maximised only at . Then whatever is, so the regularisation toward stops working. The figure shows this with hard labels. Real datasets have one label per pair, so many pairs are effectively deterministic. Azar et al., "A General Theoretical Paradigm to Understand Learning from Human Preferences" (AISTATS 2024), made this argument and proposed IPO, which regresses the log-ratio margin to a fixed target instead of pushing it to infinity:
- regularisation strength, playing the role of : a larger means a smaller target margin and a policy closer to
KTO (Ethayarajh, Xu, Muennighoff, Jurafsky, Kiela, 2024) drops pairs altogether. It needs only a binary label per response, "desirable" or "undesirable", which is much cheaper to collect. Its loss applies a Kahneman–Tversky-style value function to the implicit reward , measured against a reference point estimated from the batch's KL. Desirable and undesirable examples get separate weights to reflect loss aversion. ORPO goes further and drops the reference model, folding preference optimisation into SFT with an odds-ratio penalty.
Both log-likelihoods can fall. DPO constrains only the margin, so it is common to see fall during training along with , just more slowly. This is especially likely when and differ by only a few tokens. Pal et al. (2024, "Smaug") add a penalty that keeps the chosen likelihood from dropping below the reference (DPO-Positive). Offline data is a limit. DPO learns only from responses someone else sampled. Unlike PPO, it never sees and corrects its own outputs. Iterative or "online" DPO regenerates and relabels pairs from the current policy each round.
Use when
- +No reward model, no sampling in the loop, no value network: two models and a classification loss
- +Same population optimum as KL-constrained RLHF under Bradley–Terry
- +Stable and simple to tune; β is the main knob
- +The policy doubles as a reward model through β·log(π_θ/π_ref)
Avoid when
- −Offline: it learns only from fixed pairs and never corrects its own samples
- −Deterministic labels let margins diverge and weaken the KL constraint (the motivation for IPO)
- −Both chosen and rejected likelihoods can fall; verbosity and length bias in the data are inherited
- −Needs paired data and a reference model in memory (or precomputed reference log-probs)
Check yourself
Why does the partition function Z(x) not appear in the DPO loss?
Check yourself
A pair is already ranked correctly with implicit-reward margin r̂_w − r̂_l = 3. How strongly does it contribute to the DPO gradient, relative to a pair at the start of training?
Remember
- ✓The KL-constrained RLHF optimum is , so , and cancels in Bradley–Terry comparisons.
- ✓DPO loss: . Its gradient weights each pair by , so pairs that are ranked wrongly count most.
- ✓ (0.1 in most of the paper's experiments) sets both how far the policy may move and the log-ratio margin needed for a given loss.
- ✓DPO needs two models and four forward passes per pair, with no reward model, sampling or PPO. It is used in Zephyr, Tülu 2 and Llama 3.
- ✓With deterministic labels the margins diverge whatever β is; IPO regresses them to 1/(2τ). KTO needs only per-response good/bad labels. Watch for falling chosen log-likelihoods.