Offline preference optimisation

Three decisions: who writes the two answers, who decides which is better, and which pairs you keep.

1. Where it comes from: the RLHF goal

   Classic RLHF trains a reward model \(r(x,y)\) from the comparisons, then asks the language model to collect as much reward as possible while staying close to the reference:

   $$\max_{\pi}\;\mathbb{E}_{x,\;y\sim\pi(\cdot\mid x)}\big[r(x,y)\big] \;-\; \beta\,\mathrm{KL}\big(\pi(\cdot\mid x)\,\big\|\,\pi_{\text{ref}}(\cdot\mid x)\big)$$

   \(\max_{\pi}\)“maximise over pi”	mathFind the model \(\pi\) that makes everything after it as large as possible.
   \(\pi\)“pi”	trainedThe language model being trained, seen as a policy: for each prompt, a probability for every possible answer.
   \(x\)	dataA prompt from the dataset.
   \(y\sim\pi(\cdot\mid x)\)“y drawn from pi given x”	generatedAn answer the model itself writes for \(x\). The dot \(\cdot\) stands for “every possible answer”: \(\pi(\cdot\mid x)\) is the whole list of answer probabilities.
   \(\mathbb{E}_{x,\,y\sim\pi}\big[\;\cdot\;\big]\)	mathThe average over prompts and over the answers the model writes for them.
   \(r(x,y)\)“r”, the reward	frozenA reward model: a separate network, trained on the comparisons, that gives each answer a number, higher for better answers.
   \(\beta\)	settingThe price of moving away from the reference. The same β that appears in the DPO loss.
   \(\mathrm{KL}\big(\pi \,\|\, \pi_{\text{ref}}\big)\)“KL divergence of pi from pi ref”	mathHow different the trained model's answer probabilities are from the reference's: \(\sum_y \pi(y\mid x)\log\frac{\pi(y\mid x)}{\pi_{\text{ref}}(y\mid x)}\). It is 0 when they are identical and grows as the model drifts. The double bar \(\|\) separates the two lists.
   \(\pi_{\text{ref}}\)	frozenThe frozen copy of the starting model.

   The reward pulls toward good answers; the KL term charges for every move away from the reference. \(\beta\) is the price of moving.

2. The best model for that goal has a closed form

   $$\pi^{*}(y\mid x) = \frac{1}{Z(x)}\;\pi_{\text{ref}}(y\mid x)\;\exp\!\Big(\frac{r(x,y)}{\beta}\Big)$$

   \(\pi^{*}(y\mid x)\)“pi star”	resultThe best possible model for the goal of step 1: the probability it gives to answer \(y\). The star means “the best one”.
   \(\pi_{\text{ref}}(y\mid x)\)	frozenThe reference model's probability for the same answer: the starting point.
   \(r(x,y)\)	frozenThe answer's reward.
   \(\beta\)	settingThe price of moving. Dividing by a small β makes the rewards count more, so the best model moves further.
   \(\exp(u)\)“exp”, or \(e^u\)	math\(e \approx 2.718\) raised to the power \(u\). It is always positive, equals 1 at \(u = 0\), and grows fast: the multiplier for each answer.
   \(Z(x)\)“Z of x”, the normaliser	mathThe sum of \(\pi_{\text{ref}}(y\mid x)\,e^{r(x,y)/\beta}\) over every possible answer. Dividing by it makes the new probabilities add up to 1. It depends only on the prompt, and in practice it cannot be computed, because there are too many possible answers.

   Take the reference and multiply each answer's probability by \(e^{r/\beta}\): high-reward answers grow, low-reward ones shrink. \(Z(x)\) rescales everything so the probabilities add up to 1 again.

   two answers, both at π_ref = 0.50; rewards: A = +1.0, B = 0.0 β = 1: A: 0.50 × e¹ = 1.36 B: 0.50 × e⁰ = 0.50 Z = 1.86 → π*: A 0.73, B 0.27 β = 0.5: A: 0.50 × e² = 3.69 B: 0.50 × e⁰ = 0.50 Z = 4.19 → π*: A 0.88, B 0.12 the smaller β, the further the best model moves from the reference

3. Turn it round: the reward hides inside the model

   Solve the last formula for \(r\):

   $$r(x,y) = \beta\log\frac{\pi^{*}(y\mid x)}{\pi_{\text{ref}}(y\mid x)} + \beta\log Z(x)$$

   \(r(x,y)\)	resultThe reward that this model implies for answer \(y\).
   \(\log\frac{\pi^{*}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}\)	mathThe log-ratio: how many times more likely the model makes the answer than the reference, on a log scale. With \(\pi_\theta\) in place of \(\pi^*\) this is exactly the score \(\hat r\) of the DPO loss.
   \(\beta\)	settingThe same leash; here it converts the log-ratio into reward units.
   \(\beta\log Z(x)\)	mathA constant for each prompt: the same number for every answer to \(x\). That is why it vanishes when two answers to the same prompt are compared.

   Any model, compared with the reference, implies a reward. Apart from the \(\beta\log Z(x)\) term, which the next step cancels, that implied reward is the score \(\hat r\) in the DPO loss, times \(\beta\). It is why the DPO paper is subtitled “your language model is secretly a reward model”.

4. Compare two answers and \(Z\) cancels

   The Bradley–Terry model, the standard model of comparisons, says the chance that \(y_w\) beats \(y_l\) depends only on the difference of their rewards:

   $$P(y_w \succ y_l \mid x) = \sigma\big(r(x,y_w) - r(x,y_l)\big)$$

   \(P(\,\cdots\mid x)\)	resultA probability: here, the chance that a person or judge, shown both answers to \(x\), picks \(y_w\).
   \(y_w \succ y_l\)“y w is preferred to y l”	mathThe curly \(\succ\) means “beats” or “is preferred to”.
   \(r(x,y_w)\), \(r(x,y_l)\)	frozenThe rewards of the two answers. Only their difference matters.
   \(\sigma\)	mathThe sigmoid, \(\sigma(u) = 1/(1+e^{-u})\): turns the reward difference into a probability. Equal rewards give 0.5.

   rewards 2.0 against 0.5 → P(first wins) = σ(+1.5) = 0.82 rewards 0.5 against 0.5 → σ(0) = 0.50 (a coin flip) rewards 0.5 against 2.0 → σ(−1.5) = 0.18

   Substitute the implied reward. The awkward \(\beta\log Z(x)\) is the same for both answers to the same prompt, so it disappears from the difference. Maximum likelihood on the observed preferences, exactly as SFT does on tokens, then gives the DPO loss: no reward model to train, no reinforcement learning to run.

5. What one training step does

   $$\nabla_\theta\mathcal{L}_{\text{DPO}} = -\beta\;\underbrace{\sigma\big(\beta\,[\hat r_\theta(y_l) - \hat r_\theta(y_w)]\big)}_{\text{how wrong the ranking still is}}\;\Big[\underbrace{\nabla_\theta\log\pi_\theta(y_w\mid x)}_{\text{raise the chosen}} - \underbrace{\nabla_\theta\log\pi_\theta(y_l\mid x)}_{\text{lower the rejected}}\Big]$$

   \(\nabla_\theta\)“nabla theta”, the gradient	mathFor every weight in \(\theta\): how much the quantity after it changes when that weight is nudged up. It is a direction in which to move the weights.
   \(\nabla_\theta\mathcal{L}_{\text{DPO}}\)	resultThe direction that increases the loss. Training steps the opposite way, so every minus sign flips: the chosen answer goes up, the rejected one down.
   \(\hat r_\theta(y_w)\), \(\hat r_\theta(y_l)\)	trainedThe two scores, log-ratios against the reference, as in the loss.
   \(\sigma\big(\beta\,[\hat r_\theta(y_l) - \hat r_\theta(y_w)]\big)\)	trainedThe weight of this pair: the probability the model currently gives to the wrong ranking. Note the order: rejected minus chosen.
   \(\nabla_\theta\log\pi_\theta(y_w\mid x)\)	trainedThe SFT direction for the chosen answer: the change to the weights that makes it more likely.
   \(\nabla_\theta\log\pi_\theta(y_l\mid x)\)	trainedThe same for the rejected answer; subtracting it makes the rejected answer less likely.
   \(\beta\)	settingScales the whole step, and inside the sigmoid sets how fast the weight fades.
   \(\underbrace{\;\cdots\;}_{\text{note}}\)	mathAn underbrace is a label: it names the part above it.

   The update is SFT on the chosen answer minus SFT on the rejected one, with a weight in front. The weight is large while the model ranks the pair the wrong way and fades once the chosen answer is comfortably ahead: DPO learns most from the pairs it still gets wrong.

   scaled gap β(r̂_w − r̂_l) weight σ(−scaled gap) −3.0 0.95 ← ranked the wrong way: full push 0.0 0.50 ← where training starts, model = reference +3.0 0.05 ← already right: almost no push

6. Why the reference matters

   Without \(\pi_{\text{ref}}\), the easiest way to win a pair would be to make the rejected answer impossible, along with everything that resembles it, and to inflate the chosen one: the model would forget how to write. The reference turns the target into “more likely than you used to be”, and \(\beta\) sets the price. In the lab, train 50 steps at \(\beta = 0.1\) and then at \(\beta = 1\): the model puts about 99% of its probability on the chosen answer at the small value, against 72% at the large one, and at the small value the push is still strong.

• Length wins pairs

  Judges and people often prefer longer answers, so chosen answers are longer on average. DPO then learns that length pays. Compare the lengths of chosen and rejected answers; consider length-matched pairs or a length-normalised loss (#13).

• Both answers can lose probability

  DPO only looks at the gap. In practice the chosen answer's log-probability often falls during training too, just less than the rejected one's. Watch both, not only the margin. Mixing in a small SFT loss on the chosen answers helps, as does DPO-Positive, which penalises any fall in the chosen answer's probability (Pal et al., 2024).

• Pairs from another model learn slowly

  Pairs written by a different model sit where your model puts little probability, so the gradients are weak. Pairs sampled from the model itself, or SFT on the chosen answers first, work better.

• The labeller is the ceiling

  DPO learns the preferences it is given, mistakes included. A judge that favours a confident style trains a confidently styled model. Check the labels against people on a sample.

• The gap keeps growing

  With clean, certain preferences the loss keeps pushing the gap wider long after the ranking is right, and the leash stops holding. One or two epochs are usually enough; IPO (#13) addresses this directly.

• Silent truncation

  DPOConfig keeps max_length = 1024 tokens by default. A long prompt pushes the answers out of the window and the pair compares cut-off text, and rows whose prompt alone fills the window are dropped without notice. Raise the limit or shorten the prompts.

Two decisions: who labels each answer, and how you balance the good rows against the bad ones.

1. Prospect theory in one table

   Kahneman and Tversky found that people value outcomes as gains or losses relative to a reference point, not in absolute terms, and that both flatten out: a second £1,000 feels smaller than the first. KTO's value has the same S shape. With \(\beta = 1\) for readable numbers:

   score above the reference point r̂ − z₀ −4 −2 0 +2 +4 value of a good answer σ(r̂ − z₀) 0.02 0.12 0.50 0.88 0.98 push on it σ·(1 − σ) 0.02 0.10 0.25 0.10 0.02

   The push is largest at the reference point and fades on both sides. A good answer far above the bar barely moves. So does one hopelessly far below it, which makes KTO somewhat robust to wrong labels: an answer labelled good whose score is already far below the reference point gets little push.

2. Why a reference point, not a partner

   In DPO the rejected answer is the bar the chosen one has to clear. KTO has no partner, so the bar is the model's general drift from the reference:

   $$z_0 \;\approx\; \max\Big(0,\ \frac{1}{m}\sum_{i=1}^{m}\log\frac{\pi_\theta(y'_i\mid x_i)}{\pi_{\text{ref}}(y'_i\mid x_i)}\Big)$$

   \(z_0\)	resultThe reference point, the bar each answer's score is compared with.
   \(\approx\)“is approximately”	mathThe batch gives an estimate, not the exact value.
   \(\max(0,\ \cdot\,)\)	mathTake the larger of 0 and the average: a negative estimate is clamped to 0.
   \(m\)	mathThe number of mismatched pairs in the batch.
   \(\frac{1}{m}\sum_{i=1}^{m}\)	mathThe average over those \(m\) pairs, \(i = 1, 2, \dots, m\).
   \(x_i\)	dataThe prompt of row \(i\).
   \(y'_i\)“y prime i”	dataThe answer of another row, glued to prompt \(x_i\). The prime mark \('\) means “a different one”.
   \(\log\frac{\pi_\theta(y'_i\mid x_i)}{\pi_{\text{ref}}(y'_i\mid x_i)}\)	trainedThe score of that mismatched pair: trained log-probability minus the frozen reference's. In the batch tab: +0.12, +0.05 and −0.02, average +0.05.

   Here \(y'_i\) is another row's answer, so the pair \((x_i, y'_i)\) is unrelated: the average measures drift that has nothing to do with quality, an estimate of \(\mathrm{KL}(\pi_\theta\,\|\,\pi_{\text{ref}})\). A good answer has to rise by more than the general drift to count as a gain; a bad one earns its value by staying below the drift, rising by less than it or falling.

3. Balancing good and bad rows

   The KTO paper recommends weights that put the good side's total weight level with the bad side's, or up to one and a half times it:

   $$1 \;\le\; \frac{\lambda_D\, n_D}{\lambda_U\, n_U} \;\le\; \frac{3}{2}$$

   \(n_D\), \(n_U\)	dataHow many good (desirable) and bad (undesirable) rows the dataset has.
   \(\lambda_D\), \(\lambda_U\)	settingThe weights you give each good row and each bad row.
   \(\lambda_D\, n_D\), \(\lambda_U\, n_U\)	mathThe total weight of each side: number of rows times weight per row.
   \(1 \le \cdots \le \frac{3}{2}\)	math\(\le\) reads “at most”. The ratio of the two totals should be between 1 (level) and 1.5 (the good side up to half as heavy again).

   Earlier versions of the paper, and the warning KTOTrainer prints, use 4/3 as the upper bound.

   n_D = 200 good rows, n_U = 800 bad rows, λ_U = 1 λ_D × 200 / 800 ≥ 1 → λ_D ≥ 4 λ_D × 200 / 800 ≤ 3/2 → λ_D ≤ 6

4. Less information per row, more rows

   “B is worse than A” says which way to move between two specific answers. “B is bad” only says that B should fall. A label carries less information than a comparison, but labels are far easier to collect. The KTO paper reports matching or beating DPO at model sizes from 1B to 30B parameters, and, for a 7B model, still beating DPO after discarding up to 90% of the good answers.

• Unbalanced labels

  Real feedback is mostly complaints. Without adjusting \(\lambda_D\) and \(\lambda_U\), the model mostly learns what not to do. Check the ratio first.

• The bar needs a real batch

  \(z_0\) is measured on mismatched pairs inside each batch. Tiny batches give a noisy bar, and a batch of one gives none: TRL requires a per-device batch size above 1.

• Labels without context

  A step labelled bad because the whole session failed may have been a fine step. Label steps, not sessions, when you can.

• Not a free upgrade

  With plenty of clean pairs, try DPO as well. KTO's strength is the data pairs cannot use, not a better use of pairs.

The same pairs as DPO, with one difference: the chosen answer is now also the SFT target, so it has to be good enough to imitate.

1. Probability, odds and log odds

   p odds p/(1−p) log odds 0.10 0.11 −2.20 0.50 1.00 0.00 0.60 1.50 +0.41 0.80 4.00 +1.39 0.95 19.00 +2.94

   Log odds run from \(-\infty\) to \(+\infty\) and stretch out near 0 and 1, where a probability looks almost stuck.

2. The penalty compares the two answers' log odds

   $$\log\frac{\text{odds}_\theta(y_w)}{\text{odds}_\theta(y_l)} = \log\text{odds}_\theta(y_w) - \log\text{odds}_\theta(y_l)$$

   \(\text{odds}_\theta(y_w)\), \(\text{odds}_\theta(y_l)\)	trainedThe odds of the chosen and of the rejected answer, \(P/(1-P)\), from the model's average per-token probability. The prompt \(x\) is left out of the notation to keep it short.
   \(\log\frac{a}{b} = \log a - \log b\)	mathThe log of a ratio is a difference of logs. So the log odds ratio is just “chosen log odds minus rejected log odds”, a gap like DPO's.

   chosen 0.60, rejected 0.40: +0.405 − (−0.405) = 0.81 → penalty −log σ(0.81) = 0.37 chosen 0.80, rejected 0.20: +1.386 − (−1.386) = 2.77 → penalty −log σ(2.77) = 0.06

3. What the penalty's gradient does

   Writing \(P_w\) and \(P_l\) for the two average probabilities, and using \(\nabla\log\text{odds}(P) = \nabla\log P\,/\,(1-P)\):

   $$\nabla_\theta\Big(-\log\sigma(\log\text{OR})\Big) = -\,\sigma\big(-\log\text{OR}\big)\left[\frac{\nabla_\theta\log P_w}{1-P_w} - \frac{\nabla_\theta\log P_l}{1-P_l}\right]$$

   \(\nabla_\theta\)“nabla theta”, the gradient	mathFor every weight: how much the quantity after it changes when that weight is nudged up. Training steps the opposite way to the loss's gradient.
   \(\log\text{OR}\)	trainedThe log odds ratio, chosen log odds minus rejected log odds.
   \(\sigma(-\log\text{OR})\)	trainedThe weight of this pair: the probability the model currently gives to the wrong ranking. Near 1 while the rejected answer is ahead, near 0 once the chosen one is well ahead.
   \(P_w\), \(P_l\)	trainedThe average per-token probabilities of the chosen and the rejected answer.
   \(\nabla_\theta\log P_w\), \(\nabla_\theta\log P_l\)	trainedThe SFT directions that make each answer more likely. The minus sign in front of the rejected one turns it into “less likely”.
   \(\frac{1}{1-P}\)	mathAn amplifier that comes from the odds: it is 1.25 at \(P = 0.2\) and 5 at \(P = 0.8\), so a rejected answer the model already finds likely gets the strongest push.

   As in DPO: raise the chosen answer, lower the rejected one, harder while the ranking is poor. The \(1/(1-P)\) factors add something new: the more likely the rejected answer already is, the harder it is pushed down.

   rejected answer's average probability 0.20 0.50 0.80 factor 1 / (1 − P) 1.25 2.00 5.00 ← a likely rejected answer gets the strongest push

4. Why no reference model is needed

   DPO needs \(\pi_{\text{ref}}\) because its loss on its own can be satisfied by wrecking the model. In ORPO the SFT term keeps pulling toward the chosen answers, token by token, so the model must stay a good writer of them; the penalty only reshapes how much probability the rejected answers keep. The price is that the chosen answers must be good enough to imitate, because now they are imitated, not just preferred.

• Chosen answers are copied

  In DPO a chosen answer only has to beat the rejected one. In ORPO it is also the SFT target, so a mediocre chosen answer is learned word for word.

• An experimental API

  ORPOTrainer lives under trl.experimental and can change between releases. Pin the TRL version.

• λ trades fluency for contrast

  Too small and ORPO is just SFT; too large and the penalty swamps the SFT anchor. Sweep a few values and evaluate both answer quality and preference win-rate.

• Style comes with the chosen answers

  The SFT term pulls the model's style toward the chosen answers, whatever model you start from. If they were written by another model, expect its voice too.

The signal is the same pairs as DPO, so the choice here is about the symptom. Pick the problem you see.

1. Why DPO's gap grows forever

   \(-\log\sigma(\beta h)\) only reaches 0 as \(h \to \infty\). When a pair's preference is certain, as it is for every pair in a clean dataset, the best answer to the loss is to make the rejected answer infinitely unlikely, whatever \(\beta\) is. The leash only holds where the data itself is uncertain.

   β·h loss −log σ(βh) push σ(−βh) 0 0.69 0.50 2 0.13 0.12 5 0.0067 0.0067 10 0.000045 0.000045 ← tiny, but never zero: with enough steps the gap keeps growing

2. IPO stops at a target

   $$\frac{\partial\mathcal{L}_{\text{IPO}}}{\partial h} = 2\Big(h - \frac{1}{2\beta}\Big)$$

   \(\frac{\partial\mathcal{L}_{\text{IPO}}}{\partial h}\)“partial L-IPO over partial h”	resultThe slope of the IPO loss as the gap \(h\) changes. Training moves \(h\) against the slope: a negative slope means “widen the gap”, a positive one “narrow it”.
   \(h\)	trainedThe gap: chosen log-ratio minus rejected log-ratio.
   \(\frac{1}{2\beta}\)	settingThe target gap, set by your choice of β.
   \(2(\cdots)\)	mathThe slope of a square \(u^2\) is \(2u\): twice the distance from the target, so the pull is proportional to how far off the gap is.

   Negative below the target, so the gap is pushed up; zero at the target; positive above it, so the gap is pulled back. The leash now holds even when every label is certain.

   β = 0.1 → target gap 1 / (2 × 0.1) = 5 h = 0: slope 2 × (0 − 5) = −10 pushed up hard h = 4: slope 2 × (4 − 5) = −2 pushed up gently h = 5: slope 0 stops h = 7: slope 2 × (7 − 5) = +4 pulled back

3. Summed scores pay for length

   DPO's score is a sum over tokens, \(\sum_t \log\frac{\pi_\theta(y_t\mid\cdot)}{\pi_{\text{ref}}(y_t\mid\cdot)}\). Once training has nudged every token of the preferred style up a little, each extra token of that style adds to the score: doubling the answer doubles the score.

   per-token log-ratio after some training: +0.05 40-token answer: summed 40 × 0.05 = 2.0 average 0.05 80-token answer: summed 80 × 0.05 = 4.0 average 0.05 ← twice the score for the same quality

4. SimPO: score answers the way generation does

   SimPO's authors point out that generation, especially with beam search, ranks text by its average log-probability per token. So they use exactly that as the score, \(\frac{1}{|y|}\log\pi_\theta(y\mid x)\). No reference is needed, training runs one model, and the margin \(\gamma\) replaces DPO's zero line.

5. A map of the variants

   Variant	Score of an answer	Reference	Loss curve	Fixes
   DPO	summed log-ratio	yes	log-sigmoid	the baseline
   Length-normalised DPO	average log-ratio	yes	log-sigmoid	length bias
   IPO	summed log-ratio; per token in TRL	yes	squared, target \(1/(2\beta)\)	overfitting clean pairs
   Hinge	summed log-ratio, as in RSO and TRL; SLiC-HF's original uses plain log-probs	yes	stops at a margin	overfitting
   Robust DPO	summed log-ratio	yes	log-sigmoid, corrected for flips	noisy labels
   SimPO	average log-prob	no	log-sigmoid with margin \(\gamma\)	length bias, memory
   CPO	log-prob	no	log-sigmoid + SFT term	memory

• No reference, no leash

  SimPO and CPO keep one model in memory. SimPO has no anchor at all; CPO's SFT term ties the model to the chosen answers, as in ORPO, but not to where it started. Watch fluency and general skills, not just the win-rate.

• Each variant brings its own knob

  IPO's β sets a target gap; SimPO adds γ and needs a much larger β than DPO. Values tuned for DPO do not carry over.

• Experimental trainers

  CPOTrainer, where SimPO lives, is under trl.experimental. Pin the version.

• Fix the data first

  If the chosen answers are systematically longer, a length-normalised loss treats the symptom; length-matched pairs fix the cause.

Three jobs: find the first wrong step, write a correct one, and check it.

1. Credit: where the push lands

   The DPO gradient raises every token of the chosen answer and lowers every token of the rejected one. With two whole answers that share nothing, the correct steps of the losing answer are pushed down too.

   whole-answer pair step-level pair pushed up A1 A2 A3 A4 pushed up the fixed step 3 pushed down B1 B2 B3 B4 pushed down B3 correct steps pushed down: 2 (B1, B2) correct steps pushed down: 0 graded steps: 8 graded steps: 2

2. A shared beginning cancels

   By the chain rule, an answer's log-probability is the sum of its pieces' log-probabilities, each given what came before. If both answers start with the same steps, those steps contribute the same amount to both scores and drop out of the gap (\(x\) is left out to save space):

   $$\hat r(s_{1:k-1}\,s^{+}_k) - \hat r(s_{1:k-1}\,s^{-}_k) = \underbrace{\hat r(s_{1:k-1}) - \hat r(s_{1:k-1})}_{=\,0} + \hat r(s^{+}_k\mid s_{1:k-1}) - \hat r(s^{-}_k\mid s_{1:k-1})$$

   \(\hat r(\cdot)\)“r hat”, the score	trainedA piece of text's log-ratio, \(\log\pi_\theta - \log\pi_{\text{ref}}\): how much more likely the trained model makes it than the reference does.
   \(s_{1:k-1}\,s^{+}_k\)	dataA whole answer: the shared steps 1 to \(k-1\) followed by the correct step \(k\). Writing pieces side by side means “this, then that”.
   \(s_{1:k-1}\,s^{-}_k\)	dataThe same shared steps followed by the wrong step \(k\).
   \(\hat r(s_{1:k-1})\)	trainedThe score of the shared steps alone. It appears once on each side, so the difference is 0.
   \(\hat r(s^{\pm}_k\mid s_{1:k-1})\)	trainedThe score of step \(k\) alone, given the shared steps before it. These are the only terms left.
   \(\underbrace{\;\cdots\;}_{=\,0}\)	mathA label under the brace: the part above it equals 0.

   Step-level DPO makes the shared beginning exact by construction and puts it in the prompt, so only the step after it counts.

3. Why the first wrong step

   Cut before it and both sides would be the same correct step: no pair. Cut after it and the history already contains the mistake, so the pair can only teach a repair; the first error itself is never pushed down. The first wrong step is the only cut that targets the mistake directly. The lab's cut slider shows all three cases.

4. Fewer pairs, cleaner ones

   A solution with an error gives one pair, at its first error; a solution with no error gives none. Step-level pairs are scarcer and costlier than whole-answer pairs, but each one carries a much cleaner signal. The Step-DPO paper reports gains on maths benchmarks from about ten thousand such pairs.

• A late first error

  If the judge names a step after the real first error, the history holds an uncaught mistake and the pair teaches the wrong lesson. Check the judge's first-error labels against people.

• The chosen step must be right

  An unverified fix that is itself wrong pushes the model toward a new mistake, exactly where it is most fragile.

• Fixes in someone else's voice

  A fix written by another model sits where the model puts little probability and teaches slowly. Resampling the step from the model itself and keeping a verified one works better.

• Near-identical pairs can lower both sides

  Two completions that differ by a single number are exactly the case where DPO most often lowers the chosen side's probability too (Pal et al., 2024). Watch the chosen step's log-probability; an SFT term on the chosen step helps.

• Only some tasks have steps

  Maths, code and agent actions split cleanly into steps. Free prose does not, and a cut in the middle of a paragraph has no clear meaning.

Three decisions: where the successful session comes from, who decides success, and which sessions you pair.

1. Why the observations drop out

   The probability of a whole session multiplies the agent's choices and the environment's responses:

   $$P(\tau) = \prod_{t}\pi(a_t\mid h_t)\;P_{\text{env}}(o_t\mid h_t, a_t)$$

   \(\tau\)“tau”, a trajectory	dataA whole session: the task, then every action and observation in order.
   \(P(\tau)\)	resultThe probability of that exact session happening, agent and environment together.
   \(\prod_t\)	mathMultiply over every turn \(t\) of the session.
   \(\pi(a_t\mid h_t)\)	trainedThe agent's part: the probability of choosing action \(a_t\) given the history \(h_t\).
   \(P_{\text{env}}(o_t\mid h_t, a_t)\)“P env”	frozenThe environment's part: the probability that it answers \(o_t\) after that action. The website, the shop or the tool, never the model.

   In the score, the trained model and the reference face the same environment, so its factors are identical on the top and the bottom of the ratio and cancel:

   $$\log\frac{P_\theta(\tau)}{P_{\text{ref}}(\tau)} = \sum_t \log\frac{\pi_\theta(a_t\mid h_t)}{\pi_{\text{ref}}(a_t\mid h_t)} + \underbrace{\sum_t \log\frac{P_{\text{env}}(o_t\mid h_t,a_t)}{P_{\text{env}}(o_t\mid h_t,a_t)}}_{=\,0}$$

   \(P_\theta(\tau)\), \(P_{\text{ref}}(\tau)\)	trainedThe probability of the session with the trained agent, and with the frozen reference agent, in the same environment.
   \(\log\frac{P_\theta(\tau)}{P_{\text{ref}}(\tau)}\)	trainedThe session's score. The log of a product is a sum of logs, so it splits into one sum for the actions and one for the observations.
   \(\log\frac{\pi_\theta(a_t\mid h_t)}{\pi_{\text{ref}}(a_t\mid h_t)}\)	trainedOne action's score: the only terms that survive.
   \(\log\frac{P_{\text{env}}}{P_{\text{env}}}\)	mathThe same number on top and bottom: a ratio of 1, whose log is 0.
   \(\underbrace{\;\cdots\;}_{=\,0}\)	mathA label under the brace: the whole sum above it is 0.

   So the right score has no observation terms at all. A trainer that also runs the observation tokens through the language model adds terms \(\log\frac{\pi_\theta(o_t\mid\cdot)}{\pi_{\text{ref}}(o_t\mid\cdot)}\), the models' guesses of text the environment wrote. Those do not cancel and do not belong in the score, which is why the observations are masked.

2. Credit is spread thin

   Every action in a session gets the same pair-level weight. A long losing session with one decisive mistake spends most of its push on actions that decided nothing.

   losing session: 20 actions, 1 decisive mistake actions pushed down: 20 of which the decisive one: 1 the other 19 are punished for being in a failed session; across many pairs this averages out

3. Which tasks give a pair

   With \(N\) attempts per task at success rate \(s\), a task gives a self-made pair only if it has at least one success and at least one failure:

   $$P(\text{the task gives a pair}) = 1 - s^{N} - (1-s)^{N}$$

   \(s\)	trainedThe agent's success rate on this task: the chance that one attempt succeeds.
   \(N\)	settingHow many attempts you run per task.
   \(s^{N}\)	mathThe chance that all \(N\) attempts succeed: no failure to pair with.
   \((1-s)^{N}\)	mathThe chance that all \(N\) attempts fail: no success to pair with.
   \(1 - \cdots - \cdots\)	resultWhatever is left: the chance of at least one success and at least one failure, that is, of getting a pair.

   s = 0.10, N = 4: 1 − 0.0001 − 0.6561 = 0.34 ← most tasks are never solved: no pair s = 0.50, N = 4: 1 − 0.0625 − 0.0625 = 0.88 s = 0.95, N = 4: 1 − 0.8145 − 0.0000 = 0.19 ← most tasks never fail with an expert session for every task: 1 − sᴺ (any one failure is enough)

4. The ETO loop

   round 0 SFT on the expert sessions → agent π₀ round k explore: πₖ runs every training task pair: each failure with the expert session on that task (pairs with equal rewards are dropped) train: DPO on the new pairs, reference = πₖ, no SFT term → agent πₖ₊₁

   Each round's agent fails in new ways, so each round produces new pairs about its own current mistakes. Training on fresh failures of the current model is the idea that the iterative and online families take further.

• Good actions punished

  The losing session's sensible turns are pushed down along with its mistake. With many pairs this averages out; with few it can teach odd avoidances. When you can find the decisive step, step-level pairs (#14) are sharper.

• Mask the environment

  The observations must not be graded, and TRL's DPOTrainer grades them: it scores every token after the prompt, tool messages included. Zero their completion_mask in a custom collator.

• Long sessions, short windows

  Sessions run to many thousands of tokens. With DPOConfig's default max_length of 1,024, only the first turns of each session would be compared, the decisive later turns would be cut away, and a row whose task prompt alone fills the window would be dropped. Raise the limit, and budget the memory: every pair is two long sequences, each run through two models.

• Length shortcuts

  Failures often run long, as the agent wanders, and successes short. The model may learn to stop early. Pair sessions of similar length, or check what the pairs reward.

• Same task only

  Pairing a success on an easy task with a failure on a hard one teaches task difficulty, not behaviour.

1. Pairs of whole answers: a chosen and a rejected answer per prompt#10 DPO
2. …and the answers drift longer, the gap overfits, or there is no memory for a reference#13 SimPO, IPO
3. Single answers labelled good or bad, or a judge's per-step labels#11 KTO
4. Pairs, a base model, and one training run instead of two#12 ORPO
5. Answers made of steps, and a way to find the first wrong one#14 Step-level
6. Agent sessions where only the outcome is known#15 Trajectory-level

1. SFT on good demonstrations, from the SFT familyA SFT family
2. Sample several answers per prompt from the SFT model, then judge them: pairs for DPO, single labels for KTO
3. One preference stage, with the loss that fits the data#10 DPO#11 KTO#13 Variants
4. Optional, for multi-step tasks: add pairs cut at the first wrong step#14 Step-level
5. Optional, for agents: session pairs, and the loop of explore, pair and train#15 Trajectory
6. Evaluate: the win-rate against the SFT model, answer length, and regression checks on general skills

Instead of steps 1 to 3, ORPO (#12) does SFT and the preference stage in one run from the base model. It needs ready-made pairs, such as an existing preference set or a strong model against a weak one, because there is no SFT model to sample from yet.