# ORPO

> ORPO (Hong, Lee, Thorne 2024) aligns a base model in one stage with no reference model. The loss is the SFT NLL on the chosen response plus λ·(−log σ) of the log odds ratio between chosen and rejected, where odds = P/(1−P) and P is the length-normalised likelihood.

Source: https://www.mlfoundry.in/learn/orpo  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 14 min read · updated 2026-10-13

**ORPO:** (Odds Ratio Preference Optimization) is the preference-alignment method of Jiwoo Hong, Noah Lee and James Thorne (KAIST AI), "ORPO: Monolithic Preference Optimization without Reference Model" (EMNLP 2024; arXiv March 2024). It trains a pretrained model directly on preference pairs $(x,y_w,y_l)$ with one loss: the ordinary negative log-likelihood of the chosen response $y_w$, plus a weighted term $-\log\sigma\big(\log\tfrac{\mathrm{odds}_\theta(y_w\mid x)}{\mathrm{odds}_\theta(y_l\mid x)}\big)$ that makes the chosen response's odds larger than the rejected one's. There is no separate SFT stage and no frozen reference model.

The standard recipe has stages. **Supervised fine-tuning (SFT)** first teaches the model the chat format and the domain. Then preference optimisation, either **RLHF** with PPO or **Direct preference optimization (DPO)**, teaches it which of two answers is better. RLHF needs a reward model and PPO with four models in memory. DPO drops the reward model and the sampling, but it still needs a frozen copy of the SFT model as a reference, and it still needs the SFT stage before it.

Hong, Lee and Thorne started from a simple observation about SFT. Cross-entropy on the chosen responses raises their likelihood, but nothing in it lowers the likelihood of bad responses. The paper shows that during SFT on chosen responses, the log-probabilities of the *rejected* responses go up as well. That is expected, since chosen and rejected answers to the same prompt share format, vocabulary and most of their content. Their fix is to add a small penalty for the rejected response *to the SFT loss itself*, so that one stage learns the domain and the preference together.

## Odds and the loss

$$
P_\theta(y\mid x)=\exp\Big(\frac1m\sum_{t=1}^m\log P_\theta(y_t\mid x,y_{<t})\Big),\qquad \mathrm{odds}_\theta(y\mid x)=\frac{P_\theta(y\mid x)}{1-P_\theta(y\mid x)}
$$

- $P_\theta(y\mid x)$: **length-normalised** likelihood, the geometric mean of the $m$ token probabilities, so it lies in $(0,1)$ and does not shrink with length
- $\mathrm{odds}_\theta(y\mid x)$: how much more likely the model is to produce $y$ than not; odds 3 means $P=0.75$
- $\log\mathrm{odds}$: $=\mathrm{logit}(P)=\log P-\log(1-P)$, which ranges over all reals

$$
\mathcal L_{ORPO}=\mathbb E_{(x,y_w,y_l)}\big[\mathcal L_{SFT}+\lambda\,\mathcal L_{OR}\big],\qquad \mathcal L_{SFT}=-\log P_\theta(y_w\mid x),\qquad \mathcal L_{OR}=-\log\sigma\Big(\log\frac{\mathrm{odds}_\theta(y_w\mid x)}{\mathrm{odds}_\theta(y_l\mid x)}\Big)
$$

- $\mathcal L_{SFT}$: the usual causal-LM cross-entropy on the chosen response, averaged over its tokens
- $\mathcal L_{OR}$: log-sigmoid of the log odds ratio: near 0 once chosen odds are well above rejected odds, about $-z$ when they are well below
- $\lambda$: weight of the preference term; TRL's `ORPOConfig` exposes it as `beta`, default 0.1
- $y_w,\ y_l$: chosen (winning) and rejected (losing) responses to the same prompt x

> Figure: Top: RLHF uses three stages and four models in its PPO stage. DPO uses two stages and keeps a frozen reference. ORPO folds both into one stage with one model. Bottom: the ORPO loss for one pair. The same weights score the chosen and the rejected response. Both get length-normalised log-likelihoods. The chosen one gets plain NLL, and both enter a log-sigmoid of the log odds ratio, weighted by λ.

## Gradient analysis

The paper derives the gradient of $\mathcal L_{OR}$ and splits it into two factors. Write $z=\log\mathrm{odds}(y_w)-\log\mathrm{odds}(y_l)$. Since $\frac{d}{d\theta}\log\mathrm{odds}=\frac{d}{d\theta}\big[\log P-\log(1-P)\big]=\frac{\nabla_\theta\log P}{1-P}$, the chain rule gives:

$$
\nabla_\theta\mathcal L_{OR}=\delta(d)\cdot h(d),\qquad \delta(d)=\Big[1+\frac{\mathrm{odds}_\theta(y_w\mid x)}{\mathrm{odds}_\theta(y_l\mid x)}\Big]^{-1}=\sigma(-z),\qquad h(d)=-\frac{\nabla_\theta\log P_\theta(y_w\mid x)}{1-P_\theta(y_w\mid x)}+\frac{\nabla_\theta\log P_\theta(y_l\mid x)}{1-P_\theta(y_l\mid x)}
$$

- $\delta(d)$: a gate in $(0,1)$: close to 1 while the model prefers the rejected response, close to 0 once the chosen odds dominate
- $h(d)$: gradient descent with $h$ raises $\log P(y_w)$ and lowers $\log P(y_l)$
- $\tfrac1{1-P}$: amplifies the push on whichever response is already likely: a high-likelihood rejected response gets suppressed hardest

So the full ORPO gradient puts weight $1+\lambda\,\sigma(-z)/(1-P_w)$ on $\nabla\log P_w$ and $-\lambda\,\sigma(-z)/(1-P_l)$ on $\nabla\log P_l$. The SFT part keeps pulling the chosen response up at full strength. The odds-ratio part adds a contrast whose strength depends on how wrong the model still is and how likely the rejected answer is. It goes quiet by itself once the pair is separated.

Why odds rather than probabilities? The authors argue that a ratio of probabilities used with SFT in one stage contrasts too hard, pushing the rejected response's likelihood down too far. The algebra shows the difference: $\log\frac{\mathrm{odds}_w}{\mathrm{odds}_l}=\log\frac{P_w}{P_l}+\log\frac{1-P_l}{1-P_w}$. When $P_w>P_l$, the second term is positive, so the odds ratio already reports a clear preference at a smaller probability gap. The loss saturates without driving $P_l$ toward zero. With $P_w=0.899$ and $P_l=0.862$ the probability ratio is only 1.044, but the odds ratio is 1.436.

**From the paper**

- **Authors:** Jiwoo Hong, Noah Lee, James Thorne (KAIST AI); EMNLP 2024
- **Models:** OPT 125M, 350M and 1.3B; Phi-2 (2.7B); Llama-2 (7B); Mistral (7B)
- **Data:** Anthropic HH-RLHF and UltraFeedback (binarized); Mistral-ORPO-β used Argilla's cleaned UltraFeedback
- **Baselines:** SFT, SFT + PPO (RLHF) and SFT + DPO, against ORPO trained from the base model in one stage
- **Headline results:** up to 12.20% on AlpacaEval 2.0, 66.19% on IFEval (instruction-level loose) and 7.32 on MT-Bench, from UltraFeedback alone
- **Cost:** two forward passes per pair (chosen and rejected) through one model. DPO also needs the reference model's log-probabilities on both, and its SFT stage runs first
- **Implementations:** TRL's `ORPOTrainer` with `ORPOConfig(beta=λ)`; the authors released the Mistral-ORPO-α and Mistral-ORPO-β checkpoints

> Figure: A toy model with 8 positions, each a softmax over 6 tokens, starting uniform. The chosen and rejected responses share 6 of their 8 tokens, so both start at P = 0.167 (1/6). Gradient descent runs on SFT alone (dashed) and on ORPO (solid). SFT alone also lifts the rejected response, because its shared tokens become likelier: P_l rises to about 0.38 near step 36 and is still 0.375 at step 50 (2.25× its start). With λ = 0.5, ORPO at step 50 has P_w = 0.856 and P_l = 0.287. The odds are 5.96 and 0.40, an odds ratio of 14.83 (log 2.696), while the probability ratio is only 2.99. L_SFT = 0.155 and L_OR = 0.065, so L = 0.188. The gate σ(−z) has fallen to 0.063, so the gradient weights are only 1.220 on ∇log P_w and −0.044 on ∇log P_l. At λ = 0.1 the effect is small (P_l = 0.351), and at λ = 1 it is larger (P_l = 0.240). With 5 of 8 tokens different, both methods keep P_l low (0.115 for SFT, 0.085 for ORPO), because there is little shared text to drag it up. Toggle the heatmap to L_OR alone to see that it depends only on the gap in log odds, so every contour line runs from the corner (0, 0) to the corner (1, 1).

## Worked example: one preference pair

A chosen response of four tokens with probabilities $(0.5,0.6,0.4,0.7)$ and a rejected one with $(0.6,0.3,0.2,0.5)$, with $\lambda=0.1$. A second, high-likelihood pair shows the $1/(1-P)$ effect: chosen $(0.9,0.95,0.9,0.85)$, rejected $(0.9,0.85,0.8,0.9)$.

Both pairs computed in Python (λ = 0.1)

| quantity | pair A | pair B (both likely) |
| --- | --- | --- |
| log P_w, log P_l (mean token log-prob) | −0.6192, −1.0043 | −0.1061, −0.1491 |
| P_w, P_l | 0.5384, 0.3663 | 0.8993, 0.8615 |
| odds_w, odds_l | 1.1662, 0.5780 | 8.9310, 6.2195 |
| odds ratio (log) | 2.0176 (0.7019) | 1.4360 (0.3618) |
| probability ratio (log) | 1.4698 (0.3851) | 1.0439 (0.0430) |
| L_SFT, L_OR | 0.6192, 0.4025 | 0.1061, 0.5285 |
| L_ORPO = L_SFT + 0.1·L_OR | 0.6595 | 0.1590 |
| σ(−z) | 0.3314 | 0.4105 |
| weight on ∇log P_w, ∇log P_l | +1.0718, −0.0523 | +1.4077, −0.2964 |

**Pair A step by step**

1. Length-normalised log-likelihoods: $\log P_w=\tfrac14(\ln0.5+\ln0.6+\ln0.4+\ln0.7)=-0.6192$, so $P_w=0.5384$. Likewise $\log P_l=-1.0043$ and $P_l=0.3663$.
2. Odds: $0.5384/0.4616=1.1662$ and $0.3663/0.6337=0.5780$. The odds ratio is $2.0176$, so $z=\ln2.0176=0.7019$. The plain probability ratio would give only $\ln1.4698=0.3851$.
3. Losses: $\mathcal L_{OR}=-\ln\sigma(0.7019)=0.4025$ and $\mathcal L_{SFT}=0.6192$, so $\mathcal L_{ORPO}=0.6192+0.1\cdot0.4025=0.6595$. The SFT term dominates, as it does in practice with small $\lambda$.
4. Gradient weights: $\sigma(-z)=0.3314$. On $\nabla\log P_w$: $1+0.1\cdot0.3314/0.4616=1.0718$. On $\nabla\log P_l$: $-0.1\cdot0.3314/0.6337=-0.0523$.
5. Pair B has nearly equal, high likelihoods. $1/(1-P_l)=7.22$ instead of $1.58$, so the push down on the rejected response is $0.2964$, 5.7 times pair A's, even though the probability ratio there is only 1.044. ORPO suppresses a rejected answer hardest when the model would actually produce it.

```python
import torch, torch.nn.functional as F

def mean_logp(logits, labels, mask):
    """Length-normalised log P(y|x): mean log-prob over the response tokens (mask = 1)."""
    lp = torch.log_softmax(logits.float(), -1).gather(-1, labels[..., None]).squeeze(-1)
    return (lp * mask).sum(-1) / mask.sum(-1)

def log_odds(logp):
    """log(P / (1 − P)) from log P, stable for P close to 1."""
    return logp - torch.log1p(-torch.exp(logp))

def orpo_loss(model, chosen_ids, chosen_mask, rejected_ids, rejected_mask, lam=0.1):
    """chosen_ids/rejected_ids: prompt + response tokens; masks are 1 on response tokens only."""
    def score(ids, m):
        logits = model(input_ids=ids).logits[:, :-1]          # position t predicts token t+1
        return mean_logp(logits, ids[:, 1:], m[:, 1:])
    lw = score(chosen_ids, chosen_mask)                        # one forward pass each,
    ll = score(rejected_ids, rejected_mask)                    # no reference model
    z = log_odds(lw) - log_odds(ll)                            # log odds ratio
    l_sft = -lw                                                # NLL of the chosen response
    l_or = -F.logsigmoid(z)
    return (l_sft + lam * l_or).mean(), {"log_odds_ratio": z.detach(), "l_or": l_or.detach()}

# the loss on the pair-A token probabilities from the worked example
lw = torch.tensor([0.5, 0.6, 0.4, 0.7]).log().mean()          # -0.6192
ll = torch.tensor([0.6, 0.3, 0.2, 0.5]).log().mean()          # -1.0043
z = log_odds(lw) - log_odds(ll)                                # 0.7019 (odds ratio 2.0176)
print(-lw + 0.1 * -F.logsigmoid(z))                            # 0.6595
```

The ORPO loss in PyTorch from a causal LM's logits, plus the pair-A numbers from the table (written for PyTorch 2; the numbers in comments were computed with numpy)

> **λ is not DPO's β:** In DPO, $\beta$ scales an implicit reward measured against a reference model. In ORPO, $\lambda$ (TRL's `beta`) only weights the odds-ratio term against the SFT loss, and there is no reference at all. The values do not transfer: 0.1 is the usual starting point, and the SFT term usually dominates the loss. Watch the log odds ratio and the rejected log-likelihood during training, not just the total loss.

> **Self-regulating contrast:** The gate $\sigma(-z)$ switches the preference term off once a pair is well separated, and $1/(1-P_l)$ switches it on hardest when the model would actually say the rejected answer. Easy pairs and unlikely rejected answers cost almost nothing. Hard negatives, which the model finds plausible, get most of the gradient. In the second worked pair, the two likelihoods differ by under 4 percentage points, yet the push on the rejected response is 5.7 times larger than in the first.

> **Length normalisation matters:** ORPO's $P$ is the geometric mean of token probabilities, not the sequence probability. A raw sequence probability is tiny for any long answer, so its odds would be roughly the probability itself and the ratio would mostly compare lengths. SimPO (Meng, Xia and Chen, 2024), another reference-free method, also uses the length-normalised log-likelihood, adding a target margin. It does not include ORPO's SFT term.

**Strengths**

- One training stage from the base model: no separate SFT run
- No reference model: one model in memory and two forward passes per pair
- The gradient concentrates on pairs the model gets wrong and on plausible rejected answers
- Simple to implement: an extra log-sigmoid term on top of the SFT loss

**Limitations**

- Needs preference pairs for all of training, including the part that would otherwise be plain SFT
- Nothing ties the model to a reference, so it can drift further from the base distribution
- $\lambda$ needs tuning, and the preference signal is small next to the NLL at the usual 0.1
- Evidence is mostly at 7B scale and on instruction-following benchmarks

**Question:** Why do the ORPO authors add a penalty on the rejected response rather than relying on SFT over the chosen responses?

- A. SFT cannot be applied to chat data
- B. SFT on chosen responses also raises the likelihood of rejected responses, which share most of their tokens and style
- C. The reference model would otherwise drift
- D. Cross-entropy is undefined for long responses

**Answer:** B. SFT on chosen responses also raises the likelihood of rejected responses, which share most of their tokens and style — Cross-entropy only pulls the target up; it never pushes alternatives down specifically. Since chosen and rejected answers to one prompt overlap heavily, SFT lifts both, as the paper shows and the toy figure reproduces (P_l rises from 0.167 to 0.375). The odds-ratio term adds the missing push down.

**Question:** Two pairs have the same odds ratio, but in pair 2 the rejected response has P_l = 0.9 instead of 0.3. How does ORPO's push on log P_l compare?

- A. Identical, because only the odds ratio matters
- B. Weaker in pair 2, since the probabilities are close to 1
- C. Stronger in pair 2, by the factor (1 − 0.3)/(1 − 0.9) = 7
- D. Zero in pair 2, because the gate σ(−z) closes

**Answer:** C. Stronger in pair 2, by the factor (1 − 0.3)/(1 − 0.9) = 7 — The weight on $\nabla\log P_l$ is $-\lambda\,\sigma(-z)/(1-P_l)$. The same odds ratio means the same $z$ and the same gate. So the only change is $1/(1-P_l)$: from $1/0.7$ to $1/0.1$, seven times larger.

**Key points**

- ORPO: $\mathcal L=-\log P_\theta(y_w\mid x)+\lambda\big(-\log\sigma(\log\tfrac{\mathrm{odds}(y_w)}{\mathrm{odds}(y_l)})\big)$ with $\mathrm{odds}=P/(1-P)$ and $P$ the geometric-mean token probability.
- One stage, one model: no separate SFT run, no reference model, two forward passes per preference pair.
- Gradient $=\sigma(-z)\cdot\big[-\nabla\log P_w/(1-P_w)+\nabla\log P_l/(1-P_l)\big]$: gated off for separated pairs, amplified for likely rejected answers.
- Motivation: SFT on chosen responses also raises the rejected ones; the odds-ratio term adds a mild penalty during SFT itself.
- Mistral-ORPO-β (7B) reached 12.20% on AlpacaEval 2.0 and 7.32 on MT-Bench from UltraFeedback alone.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
