# Direct preference optimization (DPO)

> DPO (Rafailov et al., NeurIPS 2023) solves RLHF's KL-constrained objective without a reward model or RL. Since π* ∝ π_ref·exp(r/β), the reward is β·log(π/π_ref) up to a constant, and fitting Bradley–Terry with it gives the loss −log σ(β[log π/π_ref (y_w) − log π/π_ref (y_l)]).

Source: https://www.mlfoundry.in/learn/direct-preference-optimization-dpo  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 15 min read · updated 2026-10-13

**Direct preference optimization (DPO):** is the preference-tuning method of Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (NeurIPS 2023). It trains a policy $\pi_\theta$ directly on preference pairs $(x,y_w,y_l)$ with a binary cross-entropy loss on the difference of log-probability ratios against a frozen reference $\pi_{ref}$. It has no reward model, no sampling during training, and no RL. At its optimum it gives the same policy as **RLHF** with a KL penalty, under the Bradley–Terry preference model.

**RLHF** works, but it is heavy. You train a reward model, then run **Proximal policy optimization** with four networks in memory: the policy, the reference, the reward model and the value model. You also have to generate text inside the training loop. PPO for language models is notoriously sensitive to hyperparameters. Rafailov et al. noticed that the KL-regularised objective RLHF optimises has a closed-form solution. That solution can be inverted to express the reward in terms of the policy. Substituting it into the reward model's own loss gives a loss on the policy alone: the reward-modelling step *is* the policy-optimisation step.

## Derivation

$$
\max_{\pi}\ \mathbb E_{x\sim\mathcal D,\,y\sim\pi(\cdot|x)}\big[r(x,y)\big]-\beta\,\mathrm{KL}\big(\pi(\cdot|x)\,\|\,\pi_{ref}(\cdot|x)\big)
$$

- $r(x,y)$: the (unknown) reward behind human preferences
- $\pi_{ref}$: the reference policy, normally the **Supervised fine-tuning (SFT)** model
- $\beta$: strength of the KL constraint, the same role as in RLHF

Write the objective for one prompt as a single expectation and divide by $-\beta$. Maximising it is the same as minimising $\mathbb E_{y\sim\pi}\big[\log\frac{\pi(y|x)}{\pi_{ref}(y|x)\exp(r(x,y)/\beta)}\big]$. Define $Z(x)=\sum_y\pi_{ref}(y|x)\exp(r(x,y)/\beta)$ and $\pi^\ast=\pi_{ref}\,e^{r/\beta}/Z$. The objective becomes $\mathrm{KL}(\pi\|\pi^\ast)-\log Z(x)$. $Z$ does not depend on $\pi$, and a KL is zero only when the two distributions are equal, so:

$$
\pi^\ast(y|x)=\frac{1}{Z(x)}\,\pi_{ref}(y|x)\,e^{r(x,y)/\beta}\quad\Longleftrightarrow\quad r(x,y)=\beta\log\frac{\pi^\ast(y|x)}{\pi_{ref}(y|x)}+\beta\log Z(x)
$$

- $Z(x)$: a sum over every possible response, which is intractable. This is why RLHF approximates $\pi^\ast$ with PPO instead of computing it
- $\beta\log Z(x)$: depends only on the prompt, so it **cancels** in any comparison of two responses to the same prompt

Now use the Bradley–Terry model, $P(y_w\succ y_l|x)=\sigma(r(x,y_w)-r(x,y_l))$, the same one used to train RLHF reward models. Only reward differences appear, so $Z(x)$ drops out. Replace the unknown $\pi^\ast$ with the trainable $\pi_\theta$, and maximise the likelihood of the observed preferences:

$$
\mathcal L_{DPO}(\pi_\theta;\pi_{ref})=-\,\mathbb E_{(x,y_w,y_l)\sim\mathcal D}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)}-\beta\log\frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\Big)\Big]
$$

- $\hat r_\theta(x,y)=\beta\log\frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$: the **implicit reward**: the policy is its own reward model
- $\log\pi(y|x)$: the sum of token log-probabilities of the response given the prompt; one forward pass per model per response
- $y_w,\ y_l$: the preferred ("won") and dispreferred ("lost") responses

The paper also proves that nothing is lost by this reparameterisation. Every reward function is equivalent, under Bradley–Terry, to one of the form $\beta\log\frac{\pi(y|x)}{\pi_{ref}(y|x)}$ for some policy $\pi$: two rewards that differ by a function of $x$ alone induce the same preferences and the same optimal policy. Fitting a reward model and then solving the KL-constrained RL problem therefore has the same population optimum as minimising $\mathcal L_{DPO}$ directly.

> Figure: RLHF with PPO (top) fits a reward model on the preference pairs, then runs an online loop: sample from the policy, score with the RM, update with PPO under a KL penalty. That loop needs the policy, value, reference and reward networks. DPO (bottom) uses the same pairs once. It computes log-probabilities of y_w and y_l under the trainable policy and the frozen reference, forms the implicit reward margin β·[log π_θ/π_ref (y_w) − log π_θ/π_ref (y_l)], and backpropagates −log σ of it into the policy.

## What the gradient does

$$
\nabla_\theta\mathcal L_{DPO}=-\beta\,\mathbb E\Big[\underbrace{\sigma\big(\hat r_\theta(x,y_l)-\hat r_\theta(x,y_w)\big)}_{\text{weight: large when the pair is ranked wrongly}}\Big(\underbrace{\nabla_\theta\log\pi_\theta(y_w|x)}_{\text{raise }y_w}-\underbrace{\nabla_\theta\log\pi_\theta(y_l|x)}_{\text{lower }y_l}\Big)\Big]
$$

- $\sigma(\hat r_l-\hat r_w)$: 0.5 at the start (where $\pi_\theta=\pi_{ref}$), near 1 if the implicit reward ranks the pair the wrong way, near 0 once the margin is large
- $\beta$: scales both the margin inside $\sigma$ and the overall gradient

The weighting is what separates DPO from a naive "push up the good, push down the bad" (unlikelihood) objective. Without it, every pair gets pushed forever, and the paper reports that the model degenerates. With it, pairs that are already ranked correctly by a wide margin stop contributing. Learning concentrates on pairs where the implicit reward disagrees with the human label. The weight is the gradient of logistic regression, which is exactly what DPO is: logistic regression of the preference labels on the implicit reward margin.

## The role of β

$\beta$ appears in two places. It sets the target: the optimum is $\pi^\ast\propto\pi_{ref}e^{r/\beta}$, so a small $\beta$ allows a large departure from the reference and a large $\beta$ keeps the policy close. It also sets the scale: the log-ratio margin $h=\log\frac{\pi_\theta}{\pi_{ref}}(y_w)-\log\frac{\pi_\theta}{\pi_{ref}}(y_l)$ needed to reach a given loss is proportional to $1/\beta$. With $\beta=0.1$, a margin of $h=10$ nats is only $\beta h=1$ in implicit-reward units, which still leaves loss $0.313$. The paper used $\beta=0.1$ in most experiments and $\beta=0.5$ for TL;DR summarisation.

**DPO in the paper and in practice**

- **Tasks:** controlled sentiment generation on IMDb (GPT-2-large, with a sentiment classifier as the true reward), TL;DR summarisation, and single-turn dialogue on Anthropic HH (Pythia-2.8B)
- **Hyperparameters:** $\beta=0.1$ (0.5 for TL;DR), batch size 64, RMSprop at learning rate $10^{-6}$ with 150 warm-up steps
- **Reference:** the SFT model. When none existed (HH), they fine-tuned the base model on the preferred responses first ("Preferred-FT")
- **Results:** on IMDb, DPO's reward-vs-KL frontier dominated PPO's, including PPO with access to the true reward. On TL;DR, GPT-4-judged win rate against reference summaries was about 61% for DPO vs 57% for PPO at temperature 0, and DPO was more robust to sampling temperature
- **Cost per pair:** 4 forward passes ($\pi_\theta$ and $\pi_{ref}$ on $y_w$ and $y_l$), backward through $\pi_\theta$ only; $\pi_{ref}$ log-probs can be precomputed once
- **Adoption:** Zephyr-7B (DPO on UltraFeedback), Tülu 2, and Llama 3's post-training rounds (SFT, rejection sampling, DPO)

> Figure: DPO solved exactly on a toy. One prompt has six responses with hidden rewards r = (1.2, 0.8, 0.5, 0, −0.4, −1.0) and reference π_ref = (0.08, 0.12, 0.30, 0.25, 0.15, 0.10). Training uses all 15 pairs, labelled with Bradley–Terry probabilities (soft) or always for the better response (hard), and full-batch gradient descent on a tabular policy. The left panel plots the per-pair loss −log σ(βh) and the gradient weight σ(−βh) against the log-ratio margin h. At the defaults (β = 0.50, soft labels) the loss starts at log 2 = 0.6931 with every weight 0.5. After 10 steps π_θ(y1) = 0.201, and by step 400 the loss is 0.5682 and the policy equals the RLHF optimum π_ref·exp(r/β)/Z = (0.336, 0.227, 0.311, 0.095, 0.026, 0.005): TV distance 0.000, KL to the reference 0.485 nats. The implicit rewards then match the true ones up to a constant. β = 0.1 gives π(y1) = 0.970 (KL 2.365), and β = 2 gives π(y1) = 0.127 (KL 0.041). With hard labels there is no finite optimum. At β = 0.5, by step 400 the largest margin is 24.27, π(y1) = 0.993 and the KL is 2.480, approaching log(1/0.08) = 2.526, and it keeps going at every β. IPO (squared loss toward margin 1/(2β)) on the same hard labels stops at π(y1) = 0.170, KL 0.100. Margins beyond 14 are drawn at the panel edge.

## Worked example: one pair

Take $\beta=0.1$ and one pair whose summed log-probabilities are $\log\pi_\theta(y_w|x)=-12.0$ and $\log\pi_{ref}(y_w|x)=-12.5$, $\log\pi_\theta(y_l|x)=-14.0$ and $\log\pi_{ref}(y_l|x)=-13.2$.

**Loss and gradient weight**

1. Implicit rewards: $\hat r_w=0.1\cdot(-12.0+12.5)=0.05$, $\hat r_l=0.1\cdot(-14.0+13.2)=-0.08$.
2. Margin $\hat r_w-\hat r_l=0.13$. Loss $-\log\sigma(0.13)=0.6303$, slightly below the starting value $\log2=0.6931$.
3. Gradient weight $\sigma(-0.13)=0.4675$, so the update is $0.1\cdot0.4675=0.0468$ times $(\nabla\log\pi_\theta(y_w)-\nabla\log\pi_\theta(y_l))$.
4. Note that $y_l$ has *lost* 0.8 nats relative to the reference while $y_w$ gained only 0.5. DPO only sees the difference: it would be equally happy if both went down, as long as $y_l$ went down more.

Per-pair DPO loss and gradient weight as a function of the implicit-reward margin m = β·h (computed)

| m = r̂_w − r̂_l | −log σ(m) | weight σ(−m) |
| --- | --- | --- |
| −2 | 2.1269 | 0.8808 |
| −1 | 1.3133 | 0.7311 |
| 0 (start, π_θ = π_ref) | 0.6931 | 0.5000 |
| 1 | 0.3133 | 0.2689 |
| 2 | 0.1269 | 0.1192 |
| 3 | 0.0486 | 0.0474 |
| 5 | 0.0067 | 0.0067 |

```python
import torch, torch.nn.functional as F

def seq_logprob(model, input_ids, attention_mask, labels):
    """Sum of log p(token) over response tokens. labels = input_ids with prompt and padding set to -100."""
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits[:, :-1]
    tgt = labels[:, 1:]
    mask = tgt != -100
    logp = torch.gather(logits.log_softmax(-1), 2, tgt.clamp(min=0).unsqueeze(-1)).squeeze(-1)
    return (logp * mask).sum(-1)                                   # (B,)

def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1):
    """All inputs: (B,) summed log-probs of the chosen (w) and rejected (l) responses."""
    r_w = beta * (pi_w - ref_w)                                    # implicit rewards
    r_l = beta * (pi_l - ref_l)
    loss = -F.logsigmoid(r_w - r_l).mean()
    return loss, r_w.detach(), r_l.detach()

loss, r_w, r_l = dpo_loss(torch.tensor([-12.0]), torch.tensor([-14.0]),
                          torch.tensor([-12.5]), torch.tensor([-13.2]))
print(loss, r_w, r_l)          # loss 0.6303, r_w 0.05, r_l -0.08  (the worked example)

# training step (policy trainable, reference frozen, both start as the SFT model)
# with torch.no_grad():
#     ref_w = seq_logprob(ref, ids_w, mask_w, labels_w); ref_l = seq_logprob(ref, ids_l, mask_l, labels_l)
# pi_w = seq_logprob(policy, ids_w, mask_w, labels_w);  pi_l = seq_logprob(policy, ids_l, mask_l, labels_l)
# loss, r_w, r_l = dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1)
# loss.backward(); opt.step(); opt.zero_grad()
# log: reward accuracy (r_w > r_l).float().mean(), margin (r_w - r_l).mean(), and pi_w itself

def ipo_loss(pi_w, pi_l, ref_w, ref_l, tau=0.1):
    h = (pi_w - ref_w) - (pi_l - ref_l)                            # log-ratio margin
    return ((h - 1 / (2 * tau)) ** 2).mean()                       # regress to a fixed margin

# Libraries: trl.DPOTrainer implements this loss (and IPO/KTO variants) on (prompt, chosen, rejected) data.
```

DPO loss from sequence log-probabilities, with label masking for the prompt (PyTorch; written for clarity, not run here. The numbers in the comments were computed with numpy)

## Weaknesses, and IPO and KTO

**Deterministic preferences break the KL constraint.** If $y_w$ always beats $y_l$, the Bradley–Terry likelihood is maximised only at $\hat r_w-\hat r_l\to\infty$. Then $\pi_\theta(y_l)\to0$ whatever $\beta$ is, so the regularisation toward $\pi_{ref}$ stops working. The figure shows this with hard labels. Real datasets have one label per pair, so many pairs are effectively deterministic. Azar et al., "A General Theoretical Paradigm to Understand Learning from Human Preferences" (AISTATS 2024), made this argument and proposed **IPO**, which regresses the log-ratio margin to a fixed target instead of pushing it to infinity:

$$
\mathcal L_{IPO}=\mathbb E\Big[\Big(\log\frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)}-\log\frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}-\frac{1}{2\tau}\Big)^2\Big]
$$

- $\tau$: regularisation strength, playing the role of $\beta$: a larger $\tau$ means a smaller target margin and a policy closer to $\pi_{ref}$

**KTO** (Ethayarajh, Xu, Muennighoff, Jurafsky, Kiela, 2024) drops pairs altogether. It needs only a binary label per response, "desirable" or "undesirable", which is much cheaper to collect. Its loss applies a Kahneman–Tversky-style value function to the implicit reward $\beta\log\frac{\pi_\theta}{\pi_{ref}}$, measured against a reference point estimated from the batch's KL. Desirable and undesirable examples get separate weights to reflect loss aversion. **ORPO** goes further and drops the reference model, folding preference optimisation into SFT with an odds-ratio penalty.

**Both log-likelihoods can fall.** DPO constrains only the margin, so it is common to see $\log\pi_\theta(y_w)$ fall during training along with $\log\pi_\theta(y_l)$, just more slowly. This is especially likely when $y_w$ and $y_l$ differ by only a few tokens. Pal et al. (2024, "Smaug") add a penalty that keeps the chosen likelihood from dropping below the reference (DPO-Positive). **Offline data is a limit.** DPO learns only from responses someone else sampled. Unlike PPO, it never sees and corrects its own outputs. Iterative or "online" DPO regenerates and relabels pairs from the current policy each round.

> **Monitor the chosen log-probability, not just the loss:** A falling DPO loss and rising "reward accuracy" ($\hat r_w>\hat r_l$) can coexist with a model that has become less likely to produce *either* response, and that drifts toward odd text. Log $\log\pi_\theta(y_w)$ and $\log\pi_\theta(y_l)$ separately, plus the KL to the reference on fresh samples. If both log-likelihoods are falling fast, lower the learning rate (the paper used $10^{-6}$), raise $\beta$, or add an SFT or DPO-Positive term.

> **The reward model was there all along:** Under Bradley–Terry, a reward model and a policy are two parametrisations of the same object: $r=\beta\log\frac{\pi}{\pi_{ref}}+\text{const}(x)$. In the figure, with soft labels, DPO lands exactly on $\pi_{ref}e^{r/\beta}/Z$, the policy PPO would be trying to approximate, and the implicit rewards equal the true ones up to a shift. A DPO-trained model can therefore be used as a reward model: score any response by $\beta\log\frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$.

> **When RLHF still wins:** DPO is cheaper and more stable, and it is the default open-source recipe. But it is offline: the policy learns only from responses that someone else sampled. PPO samples from the current policy and can be rewarded for its own new behaviour. With a strong reward model and careful tuning, several studies report that on-policy methods (PPO, **GRPO**, online DPO) beat offline DPO, especially for reasoning and code, where rewards can be checked.

**Strengths**

- No reward model, no sampling in the loop, no value network: two models and a classification loss
- Same population optimum as KL-constrained RLHF under Bradley–Terry
- Stable and simple to tune; β is the main knob
- The policy doubles as a reward model through β·log(π_θ/π_ref)

**Limitations**

- Offline: it learns only from fixed pairs and never corrects its own samples
- Deterministic labels let margins diverge and weaken the KL constraint (the motivation for IPO)
- Both chosen and rejected likelihoods can fall; verbosity and length bias in the data are inherited
- Needs paired data and a reference model in memory (or precomputed reference log-probs)

**Question:** Why does the partition function Z(x) not appear in the DPO loss?

- A. Because it is approximated by sampling from the reference model
- B. Because the Bradley–Terry model depends only on reward differences between two responses to the same prompt, and β·log Z(x) is the same for both
- C. Because DPO assumes Z(x) = 1 for normalised policies
- D. Because DPO uses a separate reward model to estimate it

**Answer:** B. Because the Bradley–Terry model depends only on reward differences between two responses to the same prompt, and β·log Z(x) is the same for both — Inverting the optimum gives $r(x,y)=\beta\log\frac{\pi^\ast(y|x)}{\pi_{ref}(y|x)}+\beta\log Z(x)$. In $r(x,y_w)-r(x,y_l)$ the $\beta\log Z(x)$ terms cancel exactly, leaving only log-probability ratios, which a forward pass can compute.

**Question:** A pair is already ranked correctly with implicit-reward margin r̂_w − r̂_l = 3. How strongly does it contribute to the DPO gradient, relative to a pair at the start of training?

- A. Equally: every pair gets the same weight
- B. About 0.047 vs 0.5, roughly a tenth as much, because the weight is σ(r̂_l − r̂_w)
- C. Three times as much, because the margin is larger
- D. Not at all: correctly ranked pairs are skipped

**Answer:** B. About 0.047 vs 0.5, roughly a tenth as much, because the weight is σ(r̂_l − r̂_w) — The per-pair weight is $\sigma(\hat r_l-\hat r_w)=\sigma(-3)=0.0474$, compared with $\sigma(0)=0.5$ at initialisation, where $\pi_\theta=\pi_{ref}$. Correctly ranked pairs fade out smoothly but never reach exactly zero. That is why, with deterministic labels, the margins keep growing.

**Key points**

- The KL-constrained RLHF optimum is $\pi^\ast\propto\pi_{ref}e^{r/\beta}$, so $r=\beta\log\frac{\pi^\ast}{\pi_{ref}}+\beta\log Z(x)$, and $Z$ cancels in Bradley–Terry comparisons.
- DPO loss: $-\log\sigma\big(\beta\log\frac{\pi_\theta(y_w)}{\pi_{ref}(y_w)}-\beta\log\frac{\pi_\theta(y_l)}{\pi_{ref}(y_l)}\big)$. Its gradient weights each pair by $\sigma(\hat r_l-\hat r_w)$, so pairs that are ranked wrongly count most.
- $\beta$ (0.1 in most of the paper's experiments) sets both how far the policy may move and the log-ratio margin needed for a given loss.
- DPO needs two models and four forward passes per pair, with no reward model, sampling or PPO. It is used in Zephyr, Tülu 2 and Llama 3.
- With deterministic labels the margins diverge whatever β is; IPO regresses them to 1/(2τ). KTO needs only per-response good/bad labels. Watch for falling chosen log-likelihoods.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
