Direct preference optimization (DPO)

DPO (Rafailov et al., NeurIPS 2023) solves RLHF's KL-constrained objective without a reward model or RL. Since π* ∝ π_ref·exp(r/β), the reward is β·log(π/π_ref) up to a constant, and fitting Bradley–Terry with it gives the loss −log σ(β[log π/π_ref (y_w) − log π/π_ref (y_l)]).

15 min read1174 wordsadvanced0 viewsupdated 2026-10-13

Definition

Direct preference optimization (DPO) is the preference-tuning method of Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (NeurIPS 2023). It trains a policy πθ\pi_\theta directly on preference pairs (x,yw,yl)(x,y_w,y_l) with a binary cross-entropy loss on the difference of log-probability ratios against a frozen reference πref\pi_{ref}. It has no reward model, no sampling during training, and no RL. At its optimum it gives the same policy as RLHF with a KL penalty, under the Bradley–Terry preference model.

RLHF works, but it is heavy. You train a reward model, then run Proximal policy optimization with four networks in memory: the policy, the reference, the reward model and the value model. You also have to generate text inside the training loop. PPO for language models is notoriously sensitive to hyperparameters. Rafailov et al. noticed that the KL-regularised objective RLHF optimises has a closed-form solution. That solution can be inverted to express the reward in terms of the policy. Substituting it into the reward model's own loss gives a loss on the policy alone: the reward-modelling step is the policy-optimisation step.

Derivation

max⁡π Ex∼D, y∼π(⋅∣x)[r(x,y)]−β KL(π(⋅∣x) ∥ πref(⋅∣x))\max_{\pi}\ \mathbb E_{x\sim\mathcal D,\,y\sim\pi(\cdot|x)}\big[r(x,y)\big]-\beta\,\mathrm{KL}\big(\pi(\cdot|x)\,\|\,\pi_{ref}(\cdot|x)\big)
r(x,y)r(x,y)
the (unknown) reward behind human preferences
πref\pi_{ref}
the reference policy, normally the Supervised fine-tuning (SFT) model
β\beta
strength of the KL constraint, the same role as in RLHF

Write the objective for one prompt as a single expectation and divide by −β-\beta. Maximising it is the same as minimising Ey∼π[log⁡π(y∣x)πref(y∣x)exp⁡(r(x,y)/β)]\mathbb E_{y\sim\pi}\big[\log\frac{\pi(y|x)}{\pi_{ref}(y|x)\exp(r(x,y)/\beta)}\big]. Define Z(x)=∑yπref(y∣x)exp⁡(r(x,y)/β)Z(x)=\sum_y\pi_{ref}(y|x)\exp(r(x,y)/\beta) and π∗=πref er/β/Z\pi^\ast=\pi_{ref}\,e^{r/\beta}/Z. The objective becomes KL(π∥π∗)−log⁡Z(x)\mathrm{KL}(\pi\|\pi^\ast)-\log Z(x). ZZ does not depend on π\pi, and a KL is zero only when the two distributions are equal, so:

π∗(y∣x)=1Z(x) πref(y∣x) er(x,y)/β⟺r(x,y)=βlog⁡π∗(y∣x)πref(y∣x)+βlog⁡Z(x)\pi^\ast(y|x)=\frac{1}{Z(x)}\,\pi_{ref}(y|x)\,e^{r(x,y)/\beta}\quad\Longleftrightarrow\quad r(x,y)=\beta\log\frac{\pi^\ast(y|x)}{\pi_{ref}(y|x)}+\beta\log Z(x)
Z(x)Z(x)
a sum over every possible response, which is intractable. This is why RLHF approximates π∗\pi^\ast with PPO instead of computing it
βlog⁡Z(x)\beta\log Z(x)
depends only on the prompt, so it cancels in any comparison of two responses to the same prompt

Now use the Bradley–Terry model, P(yw≻yl∣x)=σ(r(x,yw)−r(x,yl))P(y_w\succ y_l|x)=\sigma(r(x,y_w)-r(x,y_l)), the same one used to train RLHF reward models. Only reward differences appear, so Z(x)Z(x) drops out. Replace the unknown π∗\pi^\ast with the trainable πθ\pi_\theta, and maximise the likelihood of the observed preferences:

LDPO(πθ;πref)=− E(x,yw,yl)∼D[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal L_{DPO}(\pi_\theta;\pi_{ref})=-\,\mathbb E_{(x,y_w,y_l)\sim\mathcal D}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)}-\beta\log\frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\Big)\Big]
r^θ(x,y)=βlog⁡πθ(y∣x)πref(y∣x)\hat r_\theta(x,y)=\beta\log\frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}
the implicit reward: the policy is its own reward model
log⁡π(y∣x)\log\pi(y|x)
the sum of token log-probabilities of the response given the prompt; one forward pass per model per response
yw, yly_w,\ y_l
the preferred ("won") and dispreferred ("lost") responses

The paper also proves that nothing is lost by this reparameterisation. Every reward function is equivalent, under Bradley–Terry, to one of the form βlog⁡π(y∣x)πref(y∣x)\beta\log\frac{\pi(y|x)}{\pi_{ref}(y|x)} for some policy π\pi: two rewards that differ by a function of xx alone induce the same preferences and the same optimal policy. Fitting a reward model and then solving the KL-constrained RL problem therefore has the same population optimum as minimising LDPO\mathcal L_{DPO} directly.

RLHF (PPO): fit a reward, then do RL against itpreferences(x, y_w, y_l)reward model r_φBradley–Terry fitsample y ~ π_θ(·|x)online, every stepPPO on r_φ − β·KLpolicy, value, ref, RMloop: generation inside trainingDPO: one supervised loss on the same pairs, no reward model, no samplingpreferences(x, y_w, y_l)policy π_θ (trained)log π_θ of y_w and y_lreference π_ref (frozen)log π_ref of y_w and y_limplicit reward marginβ[log π_θ/π_ref (y_w)− log π_θ/π_ref (y_l)]−log σ(·)mean, backpropgradients flow into π_θ onlyπ_θ and π_ref both start from the SFT model; 4 forward passes per pair (2 models × 2 answers)β·log(π_θ/π_ref) plays the role of r_φ: the language model is its own reward modelsame optimum as RLHF: π* ∝ π_ref · exp(r/β) under the Bradley–Terry model
fig.RLHF with PPO (top) fits a reward model on the preference pairs, then runs an online loop: sample from the policy, score with the RM, update with PPO under a KL penalty. That loop needs the policy, value, reference and reward networks. DPO (bottom) uses the same pairs once. It computes log-probabilities of y_w and y_l under the trainable policy and the frozen reference, forms the implicit reward margin β·[log π_θ/π_ref (y_w) − log π_θ/π_ref (y_l)], and backpropagates −log σ of it into the policy.

What the gradient does

∇θLDPO=−β E[σ(r^θ(x,yl)−r^θ(x,yw))⏟weight: large when the pair is ranked wrongly(∇θlog⁡πθ(yw∣x)⏟raise yw−∇θlog⁡πθ(yl∣x)⏟lower yl)]\nabla_\theta\mathcal L_{DPO}=-\beta\,\mathbb E\Big[\underbrace{\sigma\big(\hat r_\theta(x,y_l)-\hat r_\theta(x,y_w)\big)}_{\text{weight: large when the pair is ranked wrongly}}\Big(\underbrace{\nabla_\theta\log\pi_\theta(y_w|x)}_{\text{raise }y_w}-\underbrace{\nabla_\theta\log\pi_\theta(y_l|x)}_{\text{lower }y_l}\Big)\Big]
σ(r^l−r^w)\sigma(\hat r_l-\hat r_w)
0.5 at the start (where πθ=πref\pi_\theta=\pi_{ref}), near 1 if the implicit reward ranks the pair the wrong way, near 0 once the margin is large
β\beta
scales both the margin inside σ\sigma and the overall gradient

The weighting is what separates DPO from a naive "push up the good, push down the bad" (unlikelihood) objective. Without it, every pair gets pushed forever, and the paper reports that the model degenerates. With it, pairs that are already ranked correctly by a wide margin stop contributing. Learning concentrates on pairs where the implicit reward disagrees with the human label. The weight is the gradient of logistic regression, which is exactly what DPO is: logistic regression of the preference labels on the implicit reward margin.

The role of β

β\beta appears in two places. It sets the target: the optimum is π∗∝πrefer/β\pi^\ast\propto\pi_{ref}e^{r/\beta}, so a small β\beta allows a large departure from the reference and a large β\beta keeps the policy close. It also sets the scale: the log-ratio margin h=log⁡πθπref(yw)−log⁡πθπref(yl)h=\log\frac{\pi_\theta}{\pi_{ref}}(y_w)-\log\frac{\pi_\theta}{\pi_{ref}}(y_l) needed to reach a given loss is proportional to 1/β1/\beta. With β=0.1\beta=0.1, a margin of h=10h=10 nats is only βh=1\beta h=1 in implicit-reward units, which still leaves loss 0.3130.313. The paper used β=0.1\beta=0.1 in most experiments and β=0.5\beta=0.5 for TL;DR summarisation.

DPO in the paper and in practice

Tasks
controlled sentiment generation on IMDb (GPT-2-large, with a sentiment classifier as the true reward), TL;DR summarisation, and single-turn dialogue on Anthropic HH (Pythia-2.8B)
Hyperparameters
β=0.1\beta=0.1 (0.5 for TL;DR), batch size 64, RMSprop at learning rate 10−610^{-6} with 150 warm-up steps
Reference
the SFT model. When none existed (HH), they fine-tuned the base model on the preferred responses first ("Preferred-FT")
Results
on IMDb, DPO's reward-vs-KL frontier dominated PPO's, including PPO with access to the true reward. On TL;DR, GPT-4-judged win rate against reference summaries was about 61% for DPO vs 57% for PPO at temperature 0, and DPO was more robust to sampling temperature
Cost per pair
4 forward passes (πθ\pi_\theta and πref\pi_{ref} on ywy_w and yly_l), backward through πθ\pi_\theta only; πref\pi_{ref} log-probs can be precomputed once
Adoption
Zephyr-7B (DPO on UltraFeedback), Tülu 2, and Llama 3's post-training rounds (SFT, rejection sampling, DPO)
per-pair DPO loss vs log-ratio margin h (β = 0.50)-605101401230.51−log σ(βh)weight σ(−βh), right axis● 15 pairs (better − worse), step 400policy over 6 responsesy1y2y3y4y5y6□ π_ref · ■ π_θ · ▭ π_ref·e^(r/β)/Zcentred: ○ true r · ● β·log(π_θ/π_ref)step 400: loss 0.5682 · KL(π_θ ‖ π_ref) = 0.485 natsRLHF optimum at β = 0.50: KL(π* ‖ π_ref) = 0.485 · TV(π_θ, π*) = 0.000π_θ(y1) = 0.336 (π_ref 0.080, π* 0.336) · largest margin h = 4.40soft BT labels: DPO converges to π_ref·exp(r/β)/Z, the KL-constrained RLHF optimumtrue r = (1.2, 0.8, 0.5, 0, −0.4, −1.0) · pairs labelled i ≻ j with p = σ(r_i − r_j) (soft) or 1 (hard)
losslabels
fig.DPO solved exactly on a toy. One prompt has six responses with hidden rewards r = (1.2, 0.8, 0.5, 0, −0.4, −1.0) and reference π_ref = (0.08, 0.12, 0.30, 0.25, 0.15, 0.10). Training uses all 15 pairs, labelled with Bradley–Terry probabilities (soft) or always for the better response (hard), and full-batch gradient descent on a tabular policy. The left panel plots the per-pair loss −log σ(βh) and the gradient weight σ(−βh) against the log-ratio margin h. At the defaults (β = 0.50, soft labels) the loss starts at log 2 = 0.6931 with every weight 0.5. After 10 steps π_θ(y1) = 0.201, and by step 400 the loss is 0.5682 and the policy equals the RLHF optimum π_ref·exp(r/β)/Z = (0.336, 0.227, 0.311, 0.095, 0.026, 0.005): TV distance 0.000, KL to the reference 0.485 nats. The implicit rewards then match the true ones up to a constant. β = 0.1 gives π(y1) = 0.970 (KL 2.365), and β = 2 gives π(y1) = 0.127 (KL 0.041). With hard labels there is no finite optimum. At β = 0.5, by step 400 the largest margin is 24.27, π(y1) = 0.993 and the KL is 2.480, approaching log(1/0.08) = 2.526, and it keeps going at every β. IPO (squared loss toward margin 1/(2β)) on the same hard labels stops at π(y1) = 0.170, KL 0.100. Margins beyond 14 are drawn at the panel edge.

Worked example: one pair

Take β=0.1\beta=0.1 and one pair whose summed log-probabilities are log⁡πθ(yw∣x)=−12.0\log\pi_\theta(y_w|x)=-12.0 and log⁡πref(yw∣x)=−12.5\log\pi_{ref}(y_w|x)=-12.5, log⁡πθ(yl∣x)=−14.0\log\pi_\theta(y_l|x)=-14.0 and log⁡πref(yl∣x)=−13.2\log\pi_{ref}(y_l|x)=-13.2.

Loss and gradient weight

  1. 1Implicit rewards: r^w=0.1⋅(−12.0+12.5)=0.05\hat r_w=0.1\cdot(-12.0+12.5)=0.05, r^l=0.1⋅(−14.0+13.2)=−0.08\hat r_l=0.1\cdot(-14.0+13.2)=-0.08.
  2. 2Margin r^w−r^l=0.13\hat r_w-\hat r_l=0.13. Loss −log⁡σ(0.13)=0.6303-\log\sigma(0.13)=0.6303, slightly below the starting value log⁡2=0.6931\log2=0.6931.
  3. 3Gradient weight σ(−0.13)=0.4675\sigma(-0.13)=0.4675, so the update is 0.1⋅0.4675=0.04680.1\cdot0.4675=0.0468 times (∇log⁡πθ(yw)−∇log⁡πθ(yl))(\nabla\log\pi_\theta(y_w)-\nabla\log\pi_\theta(y_l)).
  4. 4Note that yly_l has lost 0.8 nats relative to the reference while ywy_w gained only 0.5. DPO only sees the difference: it would be equally happy if both went down, as long as yly_l went down more.
m = r̂_w − r̂_l−log σ(m)weight σ(−m)
−22.12690.8808
−11.31330.7311
0 (start, π_θ = π_ref)0.69310.5000
10.31330.2689
20.12690.1192
30.04860.0474
50.00670.0067
Per-pair DPO loss and gradient weight as a function of the implicit-reward margin m = β·h (computed)
python
1import torch, torch.nn.functional as F
2
3def seq_logprob(model, input_ids, attention_mask, labels):
4 """Sum of log p(token) over response tokens. labels = input_ids with prompt and padding set to -100."""
5 logits = model(input_ids=input_ids, attention_mask=attention_mask).logits[:, :-1]
6 tgt = labels[:, 1:]
7 mask = tgt != -100
8 logp = torch.gather(logits.log_softmax(-1), 2, tgt.clamp(min=0).unsqueeze(-1)).squeeze(-1)
9 return (logp * mask).sum(-1) # (B,)
10
11def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1):
12 """All inputs: (B,) summed log-probs of the chosen (w) and rejected (l) responses."""
13 r_w = beta * (pi_w - ref_w) # implicit rewards
14 r_l = beta * (pi_l - ref_l)
15 loss = -F.logsigmoid(r_w - r_l).mean()
16 return loss, r_w.detach(), r_l.detach()
17
18loss, r_w, r_l = dpo_loss(torch.tensor([-12.0]), torch.tensor([-14.0]),
19 torch.tensor([-12.5]), torch.tensor([-13.2]))
20print(loss, r_w, r_l) # loss 0.6303, r_w 0.05, r_l -0.08 (the worked example)
21
22# training step (policy trainable, reference frozen, both start as the SFT model)
23# with torch.no_grad():
24# ref_w = seq_logprob(ref, ids_w, mask_w, labels_w); ref_l = seq_logprob(ref, ids_l, mask_l, labels_l)
25# pi_w = seq_logprob(policy, ids_w, mask_w, labels_w); pi_l = seq_logprob(policy, ids_l, mask_l, labels_l)
26# loss, r_w, r_l = dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1)
27# loss.backward(); opt.step(); opt.zero_grad()
28# log: reward accuracy (r_w > r_l).float().mean(), margin (r_w - r_l).mean(), and pi_w itself
29
30def ipo_loss(pi_w, pi_l, ref_w, ref_l, tau=0.1):
31 h = (pi_w - ref_w) - (pi_l - ref_l) # log-ratio margin
32 return ((h - 1 / (2 * tau)) ** 2).mean() # regress to a fixed margin
33
34# Libraries: trl.DPOTrainer implements this loss (and IPO/KTO variants) on (prompt, chosen, rejected) data.
DPO loss from sequence log-probabilities, with label masking for the prompt (PyTorch; written for clarity, not run here. The numbers in the comments were computed with numpy)

Weaknesses, and IPO and KTO

Deterministic preferences break the KL constraint. If ywy_w always beats yly_l, the Bradley–Terry likelihood is maximised only at r^w−r^l→∞\hat r_w-\hat r_l\to\infty. Then πθ(yl)→0\pi_\theta(y_l)\to0 whatever β\beta is, so the regularisation toward πref\pi_{ref} stops working. The figure shows this with hard labels. Real datasets have one label per pair, so many pairs are effectively deterministic. Azar et al., "A General Theoretical Paradigm to Understand Learning from Human Preferences" (AISTATS 2024), made this argument and proposed IPO, which regresses the log-ratio margin to a fixed target instead of pushing it to infinity:

LIPO=E[(log⁡πθ(yw∣x)πref(yw∣x)−log⁡πθ(yl∣x)πref(yl∣x)−12τ)2]\mathcal L_{IPO}=\mathbb E\Big[\Big(\log\frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)}-\log\frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}-\frac{1}{2\tau}\Big)^2\Big]
τ\tau
regularisation strength, playing the role of β\beta: a larger τ\tau means a smaller target margin and a policy closer to πref\pi_{ref}

KTO (Ethayarajh, Xu, Muennighoff, Jurafsky, Kiela, 2024) drops pairs altogether. It needs only a binary label per response, "desirable" or "undesirable", which is much cheaper to collect. Its loss applies a Kahneman–Tversky-style value function to the implicit reward βlog⁡πθπref\beta\log\frac{\pi_\theta}{\pi_{ref}}, measured against a reference point estimated from the batch's KL. Desirable and undesirable examples get separate weights to reflect loss aversion. ORPO goes further and drops the reference model, folding preference optimisation into SFT with an odds-ratio penalty.

Both log-likelihoods can fall. DPO constrains only the margin, so it is common to see log⁡πθ(yw)\log\pi_\theta(y_w) fall during training along with log⁡πθ(yl)\log\pi_\theta(y_l), just more slowly. This is especially likely when ywy_w and yly_l differ by only a few tokens. Pal et al. (2024, "Smaug") add a penalty that keeps the chosen likelihood from dropping below the reference (DPO-Positive). Offline data is a limit. DPO learns only from responses someone else sampled. Unlike PPO, it never sees and corrects its own outputs. Iterative or "online" DPO regenerates and relabels pairs from the current policy each round.

Use when

  • +No reward model, no sampling in the loop, no value network: two models and a classification loss
  • +Same population optimum as KL-constrained RLHF under Bradley–Terry
  • +Stable and simple to tune; β is the main knob
  • +The policy doubles as a reward model through β·log(π_θ/π_ref)

Avoid when

  • −Offline: it learns only from fixed pairs and never corrects its own samples
  • −Deterministic labels let margins diverge and weaken the KL constraint (the motivation for IPO)
  • −Both chosen and rejected likelihoods can fall; verbosity and length bias in the data are inherited
  • −Needs paired data and a reference model in memory (or precomputed reference log-probs)

Check yourself

Why does the partition function Z(x) not appear in the DPO loss?

Check yourself

A pair is already ranked correctly with implicit-reward margin r̂_w − r̂_l = 3. How strongly does it contribute to the DPO gradient, relative to a pair at the start of training?

Remember

  • ✓The KL-constrained RLHF optimum is π∗∝πrefer/β\pi^\ast\propto\pi_{ref}e^{r/\beta}, so r=βlog⁡π∗πref+βlog⁡Z(x)r=\beta\log\frac{\pi^\ast}{\pi_{ref}}+\beta\log Z(x), and ZZ cancels in Bradley–Terry comparisons.
  • ✓DPO loss: −log⁡σ(βlog⁡πθ(yw)πref(yw)−βlog⁡πθ(yl)πref(yl))-\log\sigma\big(\beta\log\frac{\pi_\theta(y_w)}{\pi_{ref}(y_w)}-\beta\log\frac{\pi_\theta(y_l)}{\pi_{ref}(y_l)}\big). Its gradient weights each pair by σ(r^l−r^w)\sigma(\hat r_l-\hat r_w), so pairs that are ranked wrongly count most.
  • ✓β\beta (0.1 in most of the paper's experiments) sets both how far the policy may move and the log-ratio margin needed for a given loss.
  • ✓DPO needs two models and four forward passes per pair, with no reward model, sampling or PPO. It is used in Zephyr, Tülu 2 and Llama 3.
  • ✓With deterministic labels the margins diverge whatever β is; IPO regresses them to 1/(2τ). KTO needs only per-response good/bad labels. Watch for falling chosen log-likelihoods.
← back to Deep Learning roadmap