Definition
ORPO (Odds Ratio Preference Optimization) is the preference-alignment method of Jiwoo Hong, Noah Lee and James Thorne (KAIST AI), "ORPO: Monolithic Preference Optimization without Reference Model" (EMNLP 2024; arXiv March 2024). It trains a pretrained model directly on preference pairs with one loss: the ordinary negative log-likelihood of the chosen response , plus a weighted term that makes the chosen response's odds larger than the rejected one's. There is no separate SFT stage and no frozen reference model.
The standard recipe has stages. Supervised fine-tuning (SFT) first teaches the model the chat format and the domain. Then preference optimisation, either RLHF with PPO or Direct preference optimization (DPO), teaches it which of two answers is better. RLHF needs a reward model and PPO with four models in memory. DPO drops the reward model and the sampling, but it still needs a frozen copy of the SFT model as a reference, and it still needs the SFT stage before it.
Hong, Lee and Thorne started from a simple observation about SFT. Cross-entropy on the chosen responses raises their likelihood, but nothing in it lowers the likelihood of bad responses. The paper shows that during SFT on chosen responses, the log-probabilities of the rejected responses go up as well. That is expected, since chosen and rejected answers to the same prompt share format, vocabulary and most of their content. Their fix is to add a small penalty for the rejected response to the SFT loss itself, so that one stage learns the domain and the preference together.
Odds and the loss
- length-normalised likelihood, the geometric mean of the token probabilities, so it lies in and does not shrink with length
- how much more likely the model is to produce than not; odds 3 means
- , which ranges over all reals
- the usual causal-LM cross-entropy on the chosen response, averaged over its tokens
- log-sigmoid of the log odds ratio: near 0 once chosen odds are well above rejected odds, about when they are well below
- weight of the preference term; TRL's
ORPOConfigexposes it asbeta, default 0.1 - chosen (winning) and rejected (losing) responses to the same prompt x
Gradient analysis
The paper derives the gradient of and splits it into two factors. Write . Since , the chain rule gives:
- a gate in : close to 1 while the model prefers the rejected response, close to 0 once the chosen odds dominate
- gradient descent with raises and lowers
- amplifies the push on whichever response is already likely: a high-likelihood rejected response gets suppressed hardest
So the full ORPO gradient puts weight on and on . The SFT part keeps pulling the chosen response up at full strength. The odds-ratio part adds a contrast whose strength depends on how wrong the model still is and how likely the rejected answer is. It goes quiet by itself once the pair is separated.
Why odds rather than probabilities? The authors argue that a ratio of probabilities used with SFT in one stage contrasts too hard, pushing the rejected response's likelihood down too far. The algebra shows the difference: . When , the second term is positive, so the odds ratio already reports a clear preference at a smaller probability gap. The loss saturates without driving toward zero. With and the probability ratio is only 1.044, but the odds ratio is 1.436.
From the paper
- Authors
- Jiwoo Hong, Noah Lee, James Thorne (KAIST AI); EMNLP 2024
- Models
- OPT 125M, 350M and 1.3B; Phi-2 (2.7B); Llama-2 (7B); Mistral (7B)
- Data
- Anthropic HH-RLHF and UltraFeedback (binarized); Mistral-ORPO-β used Argilla's cleaned UltraFeedback
- Baselines
- SFT, SFT + PPO (RLHF) and SFT + DPO, against ORPO trained from the base model in one stage
- Headline results
- up to 12.20% on AlpacaEval 2.0, 66.19% on IFEval (instruction-level loose) and 7.32 on MT-Bench, from UltraFeedback alone
- Cost
- two forward passes per pair (chosen and rejected) through one model. DPO also needs the reference model's log-probabilities on both, and its SFT stage runs first
- Implementations
- TRL's
ORPOTrainerwithORPOConfig(beta=λ); the authors released the Mistral-ORPO-α and Mistral-ORPO-β checkpoints
Worked example: one preference pair
A chosen response of four tokens with probabilities and a rejected one with , with . A second, high-likelihood pair shows the effect: chosen , rejected .
| quantity | pair A | pair B (both likely) |
|---|---|---|
| log P_w, log P_l (mean token log-prob) | −0.6192, −1.0043 | −0.1061, −0.1491 |
| P_w, P_l | 0.5384, 0.3663 | 0.8993, 0.8615 |
| odds_w, odds_l | 1.1662, 0.5780 | 8.9310, 6.2195 |
| odds ratio (log) | 2.0176 (0.7019) | 1.4360 (0.3618) |
| probability ratio (log) | 1.4698 (0.3851) | 1.0439 (0.0430) |
| L_SFT, L_OR | 0.6192, 0.4025 | 0.1061, 0.5285 |
| L_ORPO = L_SFT + 0.1·L_OR | 0.6595 | 0.1590 |
| σ(−z) | 0.3314 | 0.4105 |
| weight on ∇log P_w, ∇log P_l | +1.0718, −0.0523 | +1.4077, −0.2964 |
Pair A step by step
- 1Length-normalised log-likelihoods: , so . Likewise and .
- 2Odds: and . The odds ratio is , so . The plain probability ratio would give only .
- 3Losses: and , so . The SFT term dominates, as it does in practice with small .
- 4Gradient weights: . On : . On : .
- 5Pair B has nearly equal, high likelihoods. instead of , so the push down on the rejected response is , 5.7 times pair A's, even though the probability ratio there is only 1.044. ORPO suppresses a rejected answer hardest when the model would actually produce it.
1import torch, torch.nn.functional as F23def mean_logp(logits, labels, mask):4 """Length-normalised log P(y|x): mean log-prob over the response tokens (mask = 1)."""5 lp = torch.log_softmax(logits.float(), -1).gather(-1, labels[..., None]).squeeze(-1)6 return (lp * mask).sum(-1) / mask.sum(-1)78def log_odds(logp):9 """log(P / (1 − P)) from log P, stable for P close to 1."""10 return logp - torch.log1p(-torch.exp(logp))1112def orpo_loss(model, chosen_ids, chosen_mask, rejected_ids, rejected_mask, lam=0.1):13 """chosen_ids/rejected_ids: prompt + response tokens; masks are 1 on response tokens only."""14 def score(ids, m):15 logits = model(input_ids=ids).logits[:, :-1] # position t predicts token t+116 return mean_logp(logits, ids[:, 1:], m[:, 1:])17 lw = score(chosen_ids, chosen_mask) # one forward pass each,18 ll = score(rejected_ids, rejected_mask) # no reference model19 z = log_odds(lw) - log_odds(ll) # log odds ratio20 l_sft = -lw # NLL of the chosen response21 l_or = -F.logsigmoid(z)22 return (l_sft + lam * l_or).mean(), {"log_odds_ratio": z.detach(), "l_or": l_or.detach()}2324# the loss on the pair-A token probabilities from the worked example25lw = torch.tensor([0.5, 0.6, 0.4, 0.7]).log().mean() # -0.619226ll = torch.tensor([0.6, 0.3, 0.2, 0.5]).log().mean() # -1.004327z = log_odds(lw) - log_odds(ll) # 0.7019 (odds ratio 2.0176)28print(-lw + 0.1 * -F.logsigmoid(z)) # 0.6595Use when
- +One training stage from the base model: no separate SFT run
- +No reference model: one model in memory and two forward passes per pair
- +The gradient concentrates on pairs the model gets wrong and on plausible rejected answers
- +Simple to implement: an extra log-sigmoid term on top of the SFT loss
Avoid when
- −Needs preference pairs for all of training, including the part that would otherwise be plain SFT
- −Nothing ties the model to a reference, so it can drift further from the base distribution
- − needs tuning, and the preference signal is small next to the NLL at the usual 0.1
- −Evidence is mostly at 7B scale and on instruction-following benchmarks
Check yourself
Why do the ORPO authors add a penalty on the rejected response rather than relying on SFT over the chosen responses?
Check yourself
Two pairs have the same odds ratio, but in pair 2 the rejected response has P_l = 0.9 instead of 0.3. How does ORPO's push on log P_l compare?
Remember
- ✓ORPO: with and the geometric-mean token probability.
- ✓One stage, one model: no separate SFT run, no reference model, two forward passes per preference pair.
- ✓Gradient : gated off for separated pairs, amplified for likely rejected answers.
- ✓Motivation: SFT on chosen responses also raises the rejected ones; the odds-ratio term adds a mild penalty during SFT itself.
- ✓Mistral-ORPO-β (7B) reached 12.20% on AlpacaEval 2.0 and 7.32 on MT-Bench from UltraFeedback alone.