ORPO

ORPO (Hong, Lee, Thorne 2024) aligns a base model in one stage with no reference model. The loss is the SFT NLL on the chosen response plus λ·(−log σ) of the log odds ratio between chosen and rejected, where odds = P/(1−P) and P is the length-normalised likelihood.

14 min read851 wordsadvanced0 viewsupdated 2026-10-13

Definition

ORPO (Odds Ratio Preference Optimization) is the preference-alignment method of Jiwoo Hong, Noah Lee and James Thorne (KAIST AI), "ORPO: Monolithic Preference Optimization without Reference Model" (EMNLP 2024; arXiv March 2024). It trains a pretrained model directly on preference pairs (x,yw,yl)(x,y_w,y_l) with one loss: the ordinary negative log-likelihood of the chosen response ywy_w, plus a weighted term −log⁡σ(log⁡oddsθ(yw∣x)oddsθ(yl∣x))-\log\sigma\big(\log\tfrac{\mathrm{odds}_\theta(y_w\mid x)}{\mathrm{odds}_\theta(y_l\mid x)}\big) that makes the chosen response's odds larger than the rejected one's. There is no separate SFT stage and no frozen reference model.

The standard recipe has stages. Supervised fine-tuning (SFT) first teaches the model the chat format and the domain. Then preference optimisation, either RLHF with PPO or Direct preference optimization (DPO), teaches it which of two answers is better. RLHF needs a reward model and PPO with four models in memory. DPO drops the reward model and the sampling, but it still needs a frozen copy of the SFT model as a reference, and it still needs the SFT stage before it.

Hong, Lee and Thorne started from a simple observation about SFT. Cross-entropy on the chosen responses raises their likelihood, but nothing in it lowers the likelihood of bad responses. The paper shows that during SFT on chosen responses, the log-probabilities of the rejected responses go up as well. That is expected, since chosen and rejected answers to the same prompt share format, vocabulary and most of their content. Their fix is to add a small penalty for the rejected response to the SFT loss itself, so that one stage learns the domain and the preference together.

Odds and the loss

Pθ(y∣x)=exp⁡(1m∑t=1mlog⁡Pθ(yt∣x,y<t)),oddsθ(y∣x)=Pθ(y∣x)1−Pθ(y∣x)P_\theta(y\mid x)=\exp\Big(\frac1m\sum_{t=1}^m\log P_\theta(y_t\mid x,y_{<t})\Big),\qquad \mathrm{odds}_\theta(y\mid x)=\frac{P_\theta(y\mid x)}{1-P_\theta(y\mid x)}
Pθ(y∣x)P_\theta(y\mid x)
length-normalised likelihood, the geometric mean of the mm token probabilities, so it lies in (0,1)(0,1) and does not shrink with length
oddsθ(y∣x)\mathrm{odds}_\theta(y\mid x)
how much more likely the model is to produce yy than not; odds 3 means P=0.75P=0.75
log⁡odds\log\mathrm{odds}
=logit(P)=log⁡P−log⁡(1−P)=\mathrm{logit}(P)=\log P-\log(1-P), which ranges over all reals
LORPO=E(x,yw,yl)[LSFT+λ LOR],LSFT=−log⁡Pθ(yw∣x),LOR=−log⁡σ(log⁡oddsθ(yw∣x)oddsθ(yl∣x))\mathcal L_{ORPO}=\mathbb E_{(x,y_w,y_l)}\big[\mathcal L_{SFT}+\lambda\,\mathcal L_{OR}\big],\qquad \mathcal L_{SFT}=-\log P_\theta(y_w\mid x),\qquad \mathcal L_{OR}=-\log\sigma\Big(\log\frac{\mathrm{odds}_\theta(y_w\mid x)}{\mathrm{odds}_\theta(y_l\mid x)}\Big)
LSFT\mathcal L_{SFT}
the usual causal-LM cross-entropy on the chosen response, averaged over its tokens
LOR\mathcal L_{OR}
log-sigmoid of the log odds ratio: near 0 once chosen odds are well above rejected odds, about −z-z when they are well below
λ\lambda
weight of the preference term; TRL's ORPOConfig exposes it as beta, default 0.1
yw, yly_w,\ y_l
chosen (winning) and rejected (losing) responses to the same prompt x
alignment pipelines, starting from a pretrained base modelRLHFSFTon demosreward modelon pairsPPOpolicy+value+RM+ref3 stages · 4 models in PPODPOSFTon demosDPO on pairspolicy + frozen ref2 stages · 2 modelsORPOORPO on pairs: SFT + odds ratioone model, no reference1 stage · 1 modelthe ORPO loss for one preference pair (x, y_w, y_l)x ⊕ y_wchosenx ⊕ y_lrejectedpolicy π_θsame weights,2 forwardpasseslog P_w = mean_tlog p(y_w,t | x, y_w,<t)log P_l = mean_tlog p(y_l,t | x, y_l,<t)L_SFT = −log P_wplain NLLL_OR = −log σ(z)z = log odds_w − log odds_lodds = P/(1 − P)×λL_ORPO =L_SFT + λ·L_ORno reference model: 2 forward passes per pair (DPO: 2 policy + 2 reference, the latter often precomputed)∇L_OR = −σ(−z)·[∇log P_w/(1 − P_w) − ∇log P_l/(1 − P_l)]: strong while the rejected odds are highMistral-ORPO-β (7B, cleaned UltraFeedback): AlpacaEval 2.0 12.20%, MT-Bench 7.32 · TRL: ORPOConfig(beta=λ)
fig.Top: RLHF uses three stages and four models in its PPO stage. DPO uses two stages and keeps a frozen reference. ORPO folds both into one stage with one model. Bottom: the ORPO loss for one pair. The same weights score the chosen and the rejected response. Both get length-normalised log-likelihoods. The chosen one gets plain NLL, and both enter a log-sigmoid of the log odds ratio, weighted by λ.

Gradient analysis

The paper derives the gradient of LOR\mathcal L_{OR} and splits it into two factors. Write z=log⁡odds(yw)−log⁡odds(yl)z=\log\mathrm{odds}(y_w)-\log\mathrm{odds}(y_l). Since ddθlog⁡odds=ddθ[log⁡P−log⁡(1−P)]=∇θlog⁡P1−P\frac{d}{d\theta}\log\mathrm{odds}=\frac{d}{d\theta}\big[\log P-\log(1-P)\big]=\frac{\nabla_\theta\log P}{1-P}, the chain rule gives:

∇θLOR=δ(d)⋅h(d),δ(d)=[1+oddsθ(yw∣x)oddsθ(yl∣x)]−1=σ(−z),h(d)=−∇θlog⁡Pθ(yw∣x)1−Pθ(yw∣x)+∇θlog⁡Pθ(yl∣x)1−Pθ(yl∣x)\nabla_\theta\mathcal L_{OR}=\delta(d)\cdot h(d),\qquad \delta(d)=\Big[1+\frac{\mathrm{odds}_\theta(y_w\mid x)}{\mathrm{odds}_\theta(y_l\mid x)}\Big]^{-1}=\sigma(-z),\qquad h(d)=-\frac{\nabla_\theta\log P_\theta(y_w\mid x)}{1-P_\theta(y_w\mid x)}+\frac{\nabla_\theta\log P_\theta(y_l\mid x)}{1-P_\theta(y_l\mid x)}
δ(d)\delta(d)
a gate in (0,1)(0,1): close to 1 while the model prefers the rejected response, close to 0 once the chosen odds dominate
h(d)h(d)
gradient descent with hh raises log⁡P(yw)\log P(y_w) and lowers log⁡P(yl)\log P(y_l)
11−P\tfrac1{1-P}
amplifies the push on whichever response is already likely: a high-likelihood rejected response gets suppressed hardest

So the full ORPO gradient puts weight 1+λ σ(−z)/(1−Pw)1+\lambda\,\sigma(-z)/(1-P_w) on ∇log⁡Pw\nabla\log P_w and −λ σ(−z)/(1−Pl)-\lambda\,\sigma(-z)/(1-P_l) on ∇log⁡Pl\nabla\log P_l. The SFT part keeps pulling the chosen response up at full strength. The odds-ratio part adds a contrast whose strength depends on how wrong the model still is and how likely the rejected answer is. It goes quiet by itself once the pair is separated.

Why odds rather than probabilities? The authors argue that a ratio of probabilities used with SFT in one stage contrasts too hard, pushing the rejected response's likelihood down too far. The algebra shows the difference: log⁡oddswoddsl=log⁡PwPl+log⁡1−Pl1−Pw\log\frac{\mathrm{odds}_w}{\mathrm{odds}_l}=\log\frac{P_w}{P_l}+\log\frac{1-P_l}{1-P_w}. When Pw>PlP_w>P_l, the second term is positive, so the odds ratio already reports a clear preference at a smaller probability gap. The loss saturates without driving PlP_l toward zero. With Pw=0.899P_w=0.899 and Pl=0.862P_l=0.862 the probability ratio is only 1.044, but the odds ratio is 1.436.

From the paper

Authors
Jiwoo Hong, Noah Lee, James Thorne (KAIST AI); EMNLP 2024
Models
OPT 125M, 350M and 1.3B; Phi-2 (2.7B); Llama-2 (7B); Mistral (7B)
Data
Anthropic HH-RLHF and UltraFeedback (binarized); Mistral-ORPO-β used Argilla's cleaned UltraFeedback
Baselines
SFT, SFT + PPO (RLHF) and SFT + DPO, against ORPO trained from the base model in one stage
Headline results
up to 12.20% on AlpacaEval 2.0, 66.19% on IFEval (instruction-level loose) and 7.32 on MT-Bench, from UltraFeedback alone
Cost
two forward passes per pair (chosen and rejected) through one model. DPO also needs the reference model's log-probabilities on both, and its SFT stage runs first
Implementations
TRL's ORPOTrainer with ORPOConfig(beta=λ); the authors released the Mistral-ORPO-α and Mistral-ORPO-β checkpoints
L = −log P_w + λ·(−log σ(log odds_w − log odds_l)), odds = P/(1 − P), P = per-token geometric meanP_w = P_l000.50.511P_w (chosen)P_l (rejected)P vs step: chosen · rejected · — ORPO · - - SFT only00.51200 stepsvs P: log odds = log P − log(1 − P) · log P00.5140-4step 50 · ORPO (λ = 0.50): P_w = 0.856, P_l = 0.287 · SFT only: P_w = 0.824, P_l = 0.375 (start 0.167)odds_w = 5.96 · odds_l = 0.40 · odds ratio = 14.83 (log 2.696) · probability ratio P_w/P_l = 2.99L_SFT = −log P_w = 0.155 · L_OR = −log σ(log OR) = 0.065 · L = L_SFT + λ·L_OR = 0.188weight on ∇log P_w: 1 + λσ(−z)/(1 − P_w) = 1.220 · on ∇log P_l: −λσ(−z)/(1 − P_l) = -0.044σ(−z) = 0.063: the odds-ratio pull fades once chosen odds dominate; 1/(1 − P) amplifies it when P is highSFT alone moved P_l from 0.167 to 0.375 (2.25×): the 6 shared tokens get likelier tootoy model: 8 positions × 6-token softmax, uniform start; rejected differs from chosen in 2 of 8 tokensheatmap: L_SFT + λ·L_OR, brighter = higher (log scale) · solid path ORPO, dashed path SFT only
heatmap
fig.A toy model with 8 positions, each a softmax over 6 tokens, starting uniform. The chosen and rejected responses share 6 of their 8 tokens, so both start at P = 0.167 (1/6). Gradient descent runs on SFT alone (dashed) and on ORPO (solid). SFT alone also lifts the rejected response, because its shared tokens become likelier: P_l rises to about 0.38 near step 36 and is still 0.375 at step 50 (2.25× its start). With λ = 0.5, ORPO at step 50 has P_w = 0.856 and P_l = 0.287. The odds are 5.96 and 0.40, an odds ratio of 14.83 (log 2.696), while the probability ratio is only 2.99. L_SFT = 0.155 and L_OR = 0.065, so L = 0.188. The gate σ(−z) has fallen to 0.063, so the gradient weights are only 1.220 on ∇log P_w and −0.044 on ∇log P_l. At λ = 0.1 the effect is small (P_l = 0.351), and at λ = 1 it is larger (P_l = 0.240). With 5 of 8 tokens different, both methods keep P_l low (0.115 for SFT, 0.085 for ORPO), because there is little shared text to drag it up. Toggle the heatmap to L_OR alone to see that it depends only on the gap in log odds, so every contour line runs from the corner (0, 0) to the corner (1, 1).

Worked example: one preference pair

A chosen response of four tokens with probabilities (0.5,0.6,0.4,0.7)(0.5,0.6,0.4,0.7) and a rejected one with (0.6,0.3,0.2,0.5)(0.6,0.3,0.2,0.5), with λ=0.1\lambda=0.1. A second, high-likelihood pair shows the 1/(1−P)1/(1-P) effect: chosen (0.9,0.95,0.9,0.85)(0.9,0.95,0.9,0.85), rejected (0.9,0.85,0.8,0.9)(0.9,0.85,0.8,0.9).

quantitypair Apair B (both likely)
log P_w, log P_l (mean token log-prob)−0.6192, −1.0043−0.1061, −0.1491
P_w, P_l0.5384, 0.36630.8993, 0.8615
odds_w, odds_l1.1662, 0.57808.9310, 6.2195
odds ratio (log)2.0176 (0.7019)1.4360 (0.3618)
probability ratio (log)1.4698 (0.3851)1.0439 (0.0430)
L_SFT, L_OR0.6192, 0.40250.1061, 0.5285
L_ORPO = L_SFT + 0.1·L_OR0.65950.1590
σ(−z)0.33140.4105
weight on ∇log P_w, ∇log P_l+1.0718, −0.0523+1.4077, −0.2964
Both pairs computed in Python (λ = 0.1)

Pair A step by step

  1. 1Length-normalised log-likelihoods: log⁡Pw=14(ln⁡0.5+ln⁡0.6+ln⁡0.4+ln⁡0.7)=−0.6192\log P_w=\tfrac14(\ln0.5+\ln0.6+\ln0.4+\ln0.7)=-0.6192, so Pw=0.5384P_w=0.5384. Likewise log⁡Pl=−1.0043\log P_l=-1.0043 and Pl=0.3663P_l=0.3663.
  2. 2Odds: 0.5384/0.4616=1.16620.5384/0.4616=1.1662 and 0.3663/0.6337=0.57800.3663/0.6337=0.5780. The odds ratio is 2.01762.0176, so z=ln⁡2.0176=0.7019z=\ln2.0176=0.7019. The plain probability ratio would give only ln⁡1.4698=0.3851\ln1.4698=0.3851.
  3. 3Losses: LOR=−ln⁡σ(0.7019)=0.4025\mathcal L_{OR}=-\ln\sigma(0.7019)=0.4025 and LSFT=0.6192\mathcal L_{SFT}=0.6192, so LORPO=0.6192+0.1⋅0.4025=0.6595\mathcal L_{ORPO}=0.6192+0.1\cdot0.4025=0.6595. The SFT term dominates, as it does in practice with small λ\lambda.
  4. 4Gradient weights: σ(−z)=0.3314\sigma(-z)=0.3314. On ∇log⁡Pw\nabla\log P_w: 1+0.1⋅0.3314/0.4616=1.07181+0.1\cdot0.3314/0.4616=1.0718. On ∇log⁡Pl\nabla\log P_l: −0.1⋅0.3314/0.6337=−0.0523-0.1\cdot0.3314/0.6337=-0.0523.
  5. 5Pair B has nearly equal, high likelihoods. 1/(1−Pl)=7.221/(1-P_l)=7.22 instead of 1.581.58, so the push down on the rejected response is 0.29640.2964, 5.7 times pair A's, even though the probability ratio there is only 1.044. ORPO suppresses a rejected answer hardest when the model would actually produce it.
python
1import torch, torch.nn.functional as F
2
3def mean_logp(logits, labels, mask):
4 """Length-normalised log P(y|x): mean log-prob over the response tokens (mask = 1)."""
5 lp = torch.log_softmax(logits.float(), -1).gather(-1, labels[..., None]).squeeze(-1)
6 return (lp * mask).sum(-1) / mask.sum(-1)
7
8def log_odds(logp):
9 """log(P / (1 − P)) from log P, stable for P close to 1."""
10 return logp - torch.log1p(-torch.exp(logp))
11
12def orpo_loss(model, chosen_ids, chosen_mask, rejected_ids, rejected_mask, lam=0.1):
13 """chosen_ids/rejected_ids: prompt + response tokens; masks are 1 on response tokens only."""
14 def score(ids, m):
15 logits = model(input_ids=ids).logits[:, :-1] # position t predicts token t+1
16 return mean_logp(logits, ids[:, 1:], m[:, 1:])
17 lw = score(chosen_ids, chosen_mask) # one forward pass each,
18 ll = score(rejected_ids, rejected_mask) # no reference model
19 z = log_odds(lw) - log_odds(ll) # log odds ratio
20 l_sft = -lw # NLL of the chosen response
21 l_or = -F.logsigmoid(z)
22 return (l_sft + lam * l_or).mean(), {"log_odds_ratio": z.detach(), "l_or": l_or.detach()}
23
24# the loss on the pair-A token probabilities from the worked example
25lw = torch.tensor([0.5, 0.6, 0.4, 0.7]).log().mean() # -0.6192
26ll = torch.tensor([0.6, 0.3, 0.2, 0.5]).log().mean() # -1.0043
27z = log_odds(lw) - log_odds(ll) # 0.7019 (odds ratio 2.0176)
28print(-lw + 0.1 * -F.logsigmoid(z)) # 0.6595
The ORPO loss in PyTorch from a causal LM's logits, plus the pair-A numbers from the table (written for PyTorch 2; the numbers in comments were computed with numpy)

Use when

  • +One training stage from the base model: no separate SFT run
  • +No reference model: one model in memory and two forward passes per pair
  • +The gradient concentrates on pairs the model gets wrong and on plausible rejected answers
  • +Simple to implement: an extra log-sigmoid term on top of the SFT loss

Avoid when

  • −Needs preference pairs for all of training, including the part that would otherwise be plain SFT
  • −Nothing ties the model to a reference, so it can drift further from the base distribution
  • −λ\lambda needs tuning, and the preference signal is small next to the NLL at the usual 0.1
  • −Evidence is mostly at 7B scale and on instruction-following benchmarks

Check yourself

Why do the ORPO authors add a penalty on the rejected response rather than relying on SFT over the chosen responses?

Check yourself

Two pairs have the same odds ratio, but in pair 2 the rejected response has P_l = 0.9 instead of 0.3. How does ORPO's push on log P_l compare?

Remember

  • ✓ORPO: L=−log⁡Pθ(yw∣x)+λ(−log⁡σ(log⁡odds(yw)odds(yl)))\mathcal L=-\log P_\theta(y_w\mid x)+\lambda\big(-\log\sigma(\log\tfrac{\mathrm{odds}(y_w)}{\mathrm{odds}(y_l)})\big) with odds=P/(1−P)\mathrm{odds}=P/(1-P) and PP the geometric-mean token probability.
  • ✓One stage, one model: no separate SFT run, no reference model, two forward passes per preference pair.
  • ✓Gradient =σ(−z)⋅[−∇log⁡Pw/(1−Pw)+∇log⁡Pl/(1−Pl)]=\sigma(-z)\cdot\big[-\nabla\log P_w/(1-P_w)+\nabla\log P_l/(1-P_l)\big]: gated off for separated pairs, amplified for likely rejected answers.
  • ✓Motivation: SFT on chosen responses also raises the rejected ones; the odds-ratio term adds a mild penalty during SFT itself.
  • ✓Mistral-ORPO-β (7B) reached 12.20% on AlpacaEval 2.0 and 7.32 on MT-Bench from UltraFeedback alone.
← back to Deep Learning roadmap