The SFT family, box by box

The base · every other technique reuses it

Plain SFT Supervised fine-tuning · behaviour cloning

Show the model a prompt and the answer you want. Grade it only on the answer, one token at a time, and nudge its weights so that answer becomes more likely. Repeat over thousands of examples.

A tiny transformer runs the forward pass, the loss, backpropagation and the weight update on screen, with every number computed live.

Changes vs plain SFT: Nothing. This is the baseline.
Fixes: Teaches format, tone and skills by example. It has all three weaknesses: W1 W2 W3
Cost: ●●●●● cheap and simple
Library: Ready: Hugging Face TRL SFTTrainer

1

The idea in plain words

A pretrained model has read a huge amount of text and can continue almost anything. What it does not know yet is how you want it to behave: answer the question, follow a format, keep a tone. SFT teaches that by example.

Each training example is a pair: a prompt and the answer you want. The model reads both in one pass. At every answer position it guesses the next token, and the loss measures how surprised it was by the target token from the training example, never by its own guess. The prompt is read for context but never graded.

“Behaviour cloning” is the same idea in the language of agents: record what an expert did in each situation, and train the model to do the same thing in the same situation.

2

See it work

Next-token trainer

Click any token to see what the model predicted at that spot. Press Train and watch the right token gain probability while the loss falls. Then switch on “grade the prompt too” to see why the prompt is masked.

3

Where the training signal comes from

Pick one option in each column to build a recipe. Every column lists the realistic ways to do that step.

4

A small example

The conversation

User · promptRewrite this so it sounds polite: “Send me the report now.”

Assistant · target answerCould you please send me the report when you have a moment?

What the model actually reads

A chat template turns the messages into one sequence with special marker tokens. Under each token: 1 if it is graded, 0 if not.

<|user|>0Rewrite0 this0 so0 it0 sounds0 polite0:0 “Send0 me0 the0 report0 now0.”0<|end|>0<|assistant|>0Could1 you1 please1 send1 me1 the1 report1 when1 you1 have1 a1 moment1?1<|end|>1

The closing <|end|> is graded too. That is how the model learns when to stop talking.

5

What a training batch looks like

#	Prompt · not graded	Target answer · graded
1	Rewrite this so it sounds polite: “Send me the report now.”	Could you please send me the report when you have a moment?
2	What is 15% of 80?	15% of 80 is 0.15 × 80 = 12.
3	Translate to Italian: “Good morning”	Buongiorno
4	System: Answer in one sentence. User: Why is the sky blue?	Sunlight scatters off air molecules, and blue light scatters the most, so the sky looks blue.
5	Suggest a name for a grey cat.	How about “Smokey”?

# sft_train.jsonl: one conversation per line
{"messages": [{"role": "user", "content": "Rewrite this so it sounds polite: \"Send me the report now.\""}, {"role": "assistant", "content": "Could you please send me the report when you have a moment?"}]}
{"messages": [{"role": "user", "content": "What is 15% of 80?"}, {"role": "assistant", "content": "15% of 80 is 0.15 × 80 = 12."}]}
{"messages": [{"role": "user", "content": "Translate to Italian: \"Good morning\""}, {"role": "assistant", "content": "Buongiorno"}]}
{"messages": [{"role": "system", "content": "Answer in one sentence."}, {"role": "user", "content": "Why is the sky blue?"}, {"role": "assistant", "content": "Sunlight scatters off air molecules, and blue light scatters the most, so the sky looks blue."}]}
{"messages": [{"role": "user", "content": "Suggest a name for a grey cat."}, {"role": "assistant", "content": "How about \"Smokey\"?"}]}

With this messages format, TRL's SFTTrainer grades every token unless you set assistant_only_loss=True (the chat template needs {% generation %} markers). With prompt / completion rows, as in #2, only the completion is graded by default.

Row 3 after tokenisation, padded to 14 positions. The ids are illustrative. -100 in labels is the PyTorch convention for “ignore this position in the loss”: it is how the mask is written in practice.

input_ids320104103304105452990137675793200732001212625571320073200032000

labels-100-100-100-100-100-100-100-100-10021262557132007-100-100

attention11111111111100

A batch stacks several rows like this one into a grid. Padding fills the short rows; attention = 0 tells the model to ignore the padding entirely.

6

The loss

$$\htmlClass{term term-1}{\mathcal{L}_{\text{SFT}}(\theta)} \;=\; \htmlClass{term term-2}{-}\;\htmlClass{term term-3}{\frac{1}{\sum_{t} m_t}}\;\htmlClass{term term-4}{\sum_{t=1}^{|s|}}\;\htmlClass{term term-5}{m_t}\;\htmlClass{term term-6}{\log}\,\htmlClass{term term-7}{p_\theta\!\left(s_t \mid s_{<t}\right)}$$

Read it aloud: “At every position of the sequence, take the log of the probability the model gave to the target token from the training example. Keep only the answer positions, add them up, divide by how many there are, and flip the sign.” Point at a row of the table to light its term up in the formula.

one row $\mathcal{L}_{\text{SFT}}(\theta)$ is the loss of one training row: one sequence $s$, its prompt and its answer. Training never steps on one row's loss alone. It first combines this number over a batch, as an average of the rows or a pool of all their answer tokens, as station 7 shows step by step.

Every symbol in the formula

Symbol	What it means
$\mathcal{L}_{\text{SFT}}(\theta)$“L-SFT of theta”	resultThe loss for one training example: a single positive number, lower is better. The $(\theta)$ says it depends on the weights: change the weights and the loss changes.
$\theta$“theta”	trainedAll of the model's weights, billions of numbers. They are the only thing training changes.
$s$	dataThe whole training example as one token sequence: the prompt followed by the answer, exactly as in the “What the GPU receives” tab above.
$\|s\|$“length of s”	mathHow many tokens $s$ has, prompt and answer together.
$t$	mathA position in the sequence: $t = 1$ is the first token, $t = \|s\|$ the last.
$s_t$“s sub t”	dataThe token at position $t$ in the training example: the target, the token the model should have written there.
$s_{<t}$“s before t”	dataEvery token before position $t$. They come from the training example, never from the model's own earlier guesses. This is called teacher forcing.
$\sum_{t=1}^{\|s\|}$“sum for t from 1 to length of s”	mathRepeat the term on the right for every position $t$ of the sequence and add the results.
$m_t$“m sub t”, the mask	dataThe loss mask: 1 on answer tokens, 0 on prompt tokens. Multiplying by 0 removes a position from the sum, so the prompt is read but never graded. In code it is the `-100` in `labels`.
$\frac{1}{\sum_{t} m_t}$“one over the sum of the mask”	math$\sum_t m_t$ counts the ones in the mask, that is the number of answer tokens. Dividing by it turns the total into an average per answer token, so a 3-token answer and a 300-token answer give losses on the same scale.
$p_\theta(s_t \mid s_{<t})$“p-theta of s-t given s before t”	trainedThe probability, between 0 and 1, that the model with weights $\theta$ gives to the target token $s_t$ after reading $s_{<t}$. The model outputs a probability for every token in its vocabulary; the loss looks up only the target's.
$\mid$“given”	mathSeparates what is being scored (left) from what the model has already seen (right).
$\log$	mathThe natural logarithm (ln, base $e \approx 2.718$). $\log 1 = 0$, $\log 0.5 = -0.69$, $\log 0.01 = -4.61$: the less likely the target, the bigger the penalty, with no upper limit as $p$ approaches 0.
$-$	mathLogs of probabilities are zero or negative. The minus sign makes the loss positive, so “better” means “smaller”, which is what optimisers minimise.

What you must supply: only rows of prompt $x$ and answer $y$. The trainer glues them into $s$ and builds the mask $m$ from where the prompt ends. Every probability $p_\theta$ is computed by the model during training.

How the loss works, step by step

Glue and mark. Prompt and answer become one sequence $s$. Each position gets $m_t = 0$ (prompt) or $m_t = 1$ (answer).
One forward pass. The model reads the whole sequence at once. At every position it outputs a probability list for the next token, computed from the real tokens before it, $s_{<t}$, never from its own guesses.
Look up the target. From each list keep one number: the probability of the token that is really there, $p_\theta(s_t \mid s_{<t})$. The rest of the list is not read by the formula.
Turn it into surprise. $-\log p$ is near 0 when the model was confident and right, and grows fast when it gave the target little probability.
Mask and average. Multiply by $m_t$ so prompt positions count zero, add up, divide by the number of answer tokens. A batch then averages this number over its rows.
Step downhill. The gradient tells every weight in $\theta$ which way to move to make those target probabilities larger. One small step, then the next batch.

Worked example: prompt “What is the capital of France?”, answer “ Paris.” plus the end marker

position token s_t m_t p_θ(s_t | s_<t) −log p counted 1 … 9 prompt tokens 0 … … no (m_t = 0) 10 " Paris" 1 0.30 1.20 yes 11 "." 1 0.60 0.51 yes 12 <|end|> 1 0.40 0.92 yes Σ m_t = 3 Σ m_t · (−log p) = 2.63 L_SFT = 2.63 / 3 = 0.88

The only way to lower this loss is to raise the probability of the example's own tokens. The formula never asks whether the example is good, which is why SFT copies bad rows as faithfully as good ones (W1).

7

One training step on the GPU

Station 6 gives the loss of one row. Training does not update the weights after each row. It takes a batch of rows, sends them through the model together on the GPU, turns them into one loss number, runs one backward pass and moves every weight once. Every other technique in this workbook runs this same step and changes only a few pieces of it; its own station 7 says which.

1BatchB rows become one grid of B × T token IDs

2Forwardevery position of every row, all at once

3Loss−log p at each answer token, averaged into one number

4Backward(p − y) at each answer token, then the chain rule; each weight adds up every position

5Updateevery weight moves once, then the next batch

One step, slope by slope

Exactly what one training step does with one row: how many slopes it makes, what each one needs, and when the weights move. Take a row whose answer has 10 tokens.

one row: prompt + 10 answer tokens → ONE training step 1 forward one pass reads the whole row: all 10 predictions at once (teacher forcing) 2 loss the average of the 10 terms: L = −(1/10) Σ log p(target) 3 backward 10 slopes at the same time, one per answer token: g_t = k (p_t − y_t) k = 1/10 each one from its own p and its own target only 4 add each weight sums the 10 contributions → 1 gradient per weight 5 update every weight moves once

weight W, old value │ the 10 slopes all come from the SAME old weights (one forward pass) ▼ gradient of W = contribution₁ + contribution₂ + … + contribution₁₀ ← one sum, nothing moves yet │ ▼ W_new = W_old − η × gradient ← one move

k = 1/10 for each token, because the loss averages over 10 answer tokens. In a batch of B rows it becomes 1/(B × 10), or 1 over the answer tokens of the whole batch when the trainer pools them (point 3 below).
“At the same time” means after the one forward pass over the whole row, not after each token. All 10 targets are already in the data, so nothing waits for anything.
Not one after the other. The weight does not move 10 times. Moving after each slope would change the weights before the next slope is measured, which is a different algorithm. Here every slope is measured on the same old weights and only their sum is used. In PyTorch, loss.backward() adds every contribution into W.grad, then optimizer.step() moves W once. Gradient accumulation keeps this: its micro-batches run one after the other, but they only add into W.grad, and the weights still move once.
Read, not graded. The prompt tokens get no slope of their own. Gradient still reaches them through attention, because the 10 answer tokens read them.
With numbers: row A of the worked example below is this step with 2 answer tokens: k = ½ each, two contributions to one weight, −0.400 + 0.016 = −0.384, one move.

1The batch: rows become one grid

The trainer takes B rows, say 32, and turns each into token IDs. Rows have different lengths, so the short ones are padded up to the longest, T. The GPU receives three grids of B × T numbers, as in the “What the GPU receives” tab of station 5: the token IDs, the attention mask (1 for a real token, 0 for padding) and the labels (a copy of the token IDs where the token is graded, −100 everywhere else; the model shifts them by one inside, so the output at each position is scored against the next token).

2Forward pass: every position at once

On one row. The prediction of token t reads only the tokens before it. All the predictions are computed in the same pass, not one after the other: during training the whole text is already known (teacher forcing), so position 10 never waits for position 9's guess. The causal mask only stops each position from looking ahead.

On the batch. The GPU stacks the vectors of all B × T positions into one table and runs each layer as one matrix multiplication on that table. Rows never mix: attention looks only inside its own row, and every other layer works on each position by itself. Out comes a probability list over the whole vocabulary at every position of every row.

What the GPU holds	Shape	An 8B model (Llama 3 8B), 32 rows of 2,048 tokens
token IDs	B × T	32 × 2,048 = 65,536 positions
the vector of each position, between layers	B × T × d	65,536 × 4,096
inside each MLP	B × T × d_MLP	65,536 × 14,336
logits: one score per word	B × T × V	65,536 × 128,256 ≈ 8.4 billion numbers
the loss	1 number	1 number

The logits alone would take about 17 GB at 2 bytes per number. Memory-efficient trainers (Liger's fused cross-entropy, torchtune's chunked loss) therefore compute the loss a chunk of positions at a time; a plain Hugging Face run builds them all, often in 32-bit, which is one reason the batch per GPU stays small.

3Loss: from each row to one number

The loss of each row with the formula of station 6: $\mathcal{L}^{(1)}, \dots, \mathcal{L}^{(B)}$.
Average them into one number: $\mathcal{L}_{\text{batch}} = \frac{1}{B}\sum_{i=1}^{B}\mathcal{L}^{(i)}$. Many trainers, Hugging Face's included, instead pool the answer tokens of all rows and divide by their total count; then a long answer weighs more than a short one.
That number is for watching training, as the loss curve. The backward pass does not use its value: it uses the formula that produced it.

4Backward pass: from one number back to every weight

What a slope is. A slope, or gradient, answers one question about a number: if it rose by a tiny amount ε, how much would the loss change? A slope of −0.4 means the loss would fall by 0.4 × ε, so the update raises that number. The slope of a score and the gradient of a weight are the same idea.

Start at 1. The backward pass starts at the loss itself, whose slope with respect to itself is 1, and from there asks each earlier number: if you moved a little, how much would the loss move?
The average splits the 1. Each row gets 1/B; inside a row, each answer token gets 1/Σm. Prompt and padding positions get 0.
At each answer token: p − y. Taking the softmax and then −log, one link of the chain rule, turns that share into a slope on every word's score: $g_{i,t} = \frac{1}{B\,\sum_t m_{i,t}}\,(p_{i,t} - y_{i,t})$. Here $y_{i,t}$ is the one-hot list of the target: 1 at the target token and 0 at every other word of the vocabulary (not the answer y of the data), and $\sum_t m_{i,t}$ is the number of answer tokens of row i; with token pooling (point 3) the factor is $1/N_{\text{tokens}}$ instead, the answer-token count of the whole batch. The slope is negative for the target (raise its score) and positive for every other word (lower it). Each position's slope comes from its own p and y only.
Down through the layers. The chain rule carries each token's slope down through the output layer, the MLPs, attention and the embeddings, with the same boxes in every technique. Through attention, a graded position also sends gradient to the earlier positions it read, prompt included.
Into the weights: add up every use. A weight is used at every position of every row, so its gradient is the sum, over all of them, of the input it multiplied × the slope that came back. On the GPU that sum is one matrix multiplication per layer, $\nabla W = X^{\top} G$: X holds the inputs of all B × T positions, kept from the forward pass, and G their slopes.

$$\frac{\partial \mathcal{L}_{\text{batch}}}{\partial W} \;=\; \sum_{i=1}^{B}\,\sum_{t}\; \underbrace{x_{i,t}^{\top}}_{\text{input, kept from the forward pass}}\;\underbrace{g_{i,t}}_{\text{slope, from the backward pass}}$$

For one weight W[a][b] this is $\sum_{i,t} x_{i,t}[a]\; g_{i,t}[b]$: exactly the five-line sum of the worked example below.

5Update: every weight moves once

With one gradient per weight, the optimiser moves every weight once, all in parallel: $\theta \leftarrow \theta - \eta\, g$ in its simplest form. Real trainers use AdamW: it moves each weight along a running average of its recent gradients, divided by their typical size, so every weight takes steps of similar size, and it also shrinks each weight slightly (weight decay). Trainers also shrink the gradient when it is unusually large (clipping) and change $\eta$ on a schedule, a short warm-up and then a slow decrease. Then the gradients are reset to zero and the next batch starts. One batch, one update.

Worked example: one full step on a model you can compute by hand

A vocabulary of six words. At every position the layers below produce a vector h of 2 numbers; the output layer W_U, 2 × 6 = 12 weights, turns h into six scores, and softmax turns the scores into probabilities. To keep it short, h is given and only W_U is trained; the layers below would follow the same rules (point 4).

W_U cat dog sat on mat <end> feature 1 0.5 0.2 −0.2 0.1 0.3 −0.4 feature 2 −0.3 −0.1 0.4 0.2 0.1 0.6 ROW A prompt "cat sat on" answer "mat <end>" Σ m = 2 forward: every position at once (t is the position of the target, as in station 6) t has read h target p(target) −log p m 2 cat [0.8, −0.2] sat 0.121 2.112 0 3 cat sat [−0.5, 0.7] on 0.161 1.829 0 4 cat sat on [1.0, 0.5] mat 0.199 1.614 1 5 cat sat on mat [0.2, 1.0] <end> 0.231 1.467 1 at t = 4: z(mat) = 1.0 × 0.3 + 0.5 × 0.1 = 0.35, and softmax over the six scores gives 0.199 loss L_A = (1.614 + 1.467) / 2 = 1.54 backward slope on the six scores at each answer token: g = ½ (p − y) t = 4 (target mat): cat +0.100 dog +0.082 sat +0.070 on +0.086 mat −0.400 <end> +0.063 t = 5 (target <end>): cat +0.056 dog +0.065 sat +0.098 on +0.085 mat +0.080 <end> −0.385 t = 2, 3 (prompt): every slope 0 one weight, feature 1 → mat: Σ over positions of h[1] × g(mat) t = 4: 1.0 × (−0.400) = −0.400 mat is the target here: raise this weight t = 5: 0.2 × (+0.080) = +0.016 mat is wrong here: lower it a little gradient = −0.384 update (η = 1) 0.3 − 1 × (−0.384) = 0.684 the other 11 weights are updated by the same rule, each by its own gradient forward again p(mat) 0.199 → 0.299 p(<end>) 0.231 → 0.312 L_A 1.54 → 1.19

ROW B prompt "dog sat" answer "on mat <end>" Σ m = 3 t has read h target p(target) −log p m 2 dog [0.6, 0.1] sat 0.143 1.948 0 3 dog sat [−0.4, 0.8] on 0.162 1.822 1 4 dog sat on [0.9, 0.3] mat 0.196 1.629 1 5 dog sat on mat [0.1, 0.9] <end> 0.231 1.464 1 L_B = (1.822 + 1.629 + 1.464) / 3 = 1.64 A BATCH OF ROWS A AND B loss L = (1.54 + 1.64) / 2 = 1.59 shares row A: ½ × ½ = ¼ per answer token row B: ½ × ⅓ = ⅙ per answer token one weight, feature 1 → mat, from all five answer positions: A, t = 4 1.0 × (−0.200) = −0.200 A, t = 5 0.2 × (+0.040) = +0.008 B, t = 3 −0.4 × (+0.023) = −0.009 B, t = 4 0.9 × (−0.134) = −0.121 B, t = 5 0.1 × (+0.026) = +0.003 gradient = −0.319 one number, from both rows update once 0.3 − (−0.319) = 0.619 forward again L_A 1.54 → 1.27 L_B 1.64 → 1.46 L 1.59 → 1.37

With row A alone, the weight went to 0.684; in the batch, to 0.619. Row A's pushes were halved to share the step with row B, and row B's were added in.

At real scale

In GPU memory during one step	Per weight	An 8B model, full fine-tuning
the weights (bf16)	2 bytes	16 GB
their gradients	2 bytes	16 GB
AdamW: a full-precision copy of the weights and two running averages	12 bytes	96 GB
activations kept for the backward pass	grows with B × T	hundreds of GB; about 20–40 GB with activation checkpointing
total	about 16 bytes, plus activations	about 130 GB, plus activations

Memory. That is more than one 80 GB GPU holds, so a full fine-tune of an 8B model is spread over several GPUs. LoRA avoids most of it by training a small add-on: only the add-on has gradients and optimiser state.
Time. About 6 operations per weight per token: 2 in the forward pass and 4 in the backward pass. For 8 billion weights and 65,536 tokens that is about 3 × 10¹⁵ operations, around 3 seconds on one H100 at full speed and in practice 2 to 3 times longer. The backward pass takes about two thirds of it.
Gradient accumulation. When a batch does not fit, it is split into k micro-batches: forward and backward on each, with each micro-batch's loss first scaled by its share (1/k for an average over rows; divided by the whole batch's answer-token count when tokens are pooled), gradients added up, one update at the end. Then the result is the same as one big batch.
Activation checkpointing. Keep only some activations and recompute the rest during the backward pass: much less memory for about a third more arithmetic.
Many GPUs (data parallel). Each GPU holds a copy of the model and takes a slice of the batch, for example 8 GPUs × 4 rows. Each runs forward and backward on its slice; then the GPUs average their gradients in one exchange (an all-reduce) and every GPU applies the same update, so the copies stay identical. Sharded versions (FSDP, ZeRO) also split the weights, gradients and optimiser state across the GPUs to save memory.

8

The maths behind it

The probability of a whole answer is a product

The model writes one token at a time, so the probability of a full answer is the probability of each token multiplied together, each one given everything before it. This is the chain rule of probability.

$$p_\theta(y \mid x) = \prod_{t=1}^{|y|} p_\theta\big(y_t \mid x,\, y_{<t}\big)$$

$p_\theta(y \mid x)$	resultThe probability that the model, with weights $\theta$, writes the whole answer $y$ after the prompt $x$.
$x$, $y$	dataThe prompt and the answer of one training example.
$\prod_{t=1}^{\|y\|}$“product for t from 1 to length of y”	mathLike $\sum$, but multiplies the terms instead of adding them: one factor per answer token.
$y_t$, $y_{<t}$	dataThe answer token at position $t$, and the answer tokens before it.
$p_\theta(y_t \mid x, y_{<t})$	trainedThe probability of the next token alone, given the prompt and the answer so far. Each factor sees one more token of history than the one before.

answer " Paris." = " Paris" then "." p(" Paris" | question) = 0.30 p("." | question, " Paris") = 0.60 p(whole answer) = 0.30 × 0.60 = 0.18 ← like two coin flips: 0.5 × 0.5 = 0.25

Maximum likelihood: choose the weights that make the data most probable

Maximum likelihood answers one question: which setting of the model makes the data you actually saw the least surprising? The clearest way in is a model with a single knob.

a · The coin: a model with one knob

Imagine a “coin model” with one knob, $h$ = the chance of heads. You toss a real coin 10 times and see 7 heads and 3 tails. You do not know $h$, so for each possible value you ask: if the knob were set here, how likely would this result have been? Each head has probability $h$ and each tail $1-h$. The tosses are independent, so you multiply:

h = 0.5: 0.5 × 0.5 × … (10 times) = 0.5¹⁰ = 0.00098 h = 0.7: 0.7 × … (7 heads) × 0.3 × 0.3 × 0.3 (3 tails) = 0.7⁷ × 0.3³ = 0.00222 ← highest h = 0.9: 0.9 × … (7 heads) × 0.1 × 0.1 × 0.1 (3 tails) = 0.9⁷ × 0.1³ = 0.00048

All three numbers are tiny because any one particular sequence of 10 tosses is unlikely. Only the comparison matters:

h = 0.9 expects almost all heads, so each of the 3 tails costs it a factor of 0.1. Heavily punished.
h = 0.5 expects about 5 heads, so seeing 7 is a stretch.
h = 0.7 fits best. Try every value and the curve peaks exactly at 7/10.

These are probabilities of this exact order of heads and tails. Counting every order would multiply all three by the same 120, so the winner does not change.

b · Try it

Find the best knob

Move the knob $h$ and watch how likely the 10 tosses become. Change what the coin showed and the peak moves with it. Then press Take a training step to climb to the peak the way training does.

c · The same idea in SFT

SFT asks the same question, scaled up.

Coin	SFT
one knob, $h$	billions of knobs: the weights $\theta$
the tosses you saw	the answer tokens in your dataset
chance of heads, $h$	the probability the model gives to each correct token
“how likely is 7 heads and 3 tails?”	“how likely is every answer in the dataset?”
best knob $h^\star = 0.7$	best weights $\theta^\star$

d · Reading the formula piece by piece

$$\htmlClass{term term-1}{\theta^\star} = \htmlClass{term term-2}{\operatorname*{arg\,max}_\theta}\;\htmlClass{term term-3}{\sum_{(x,y)\in\mathcal{D}}}\;\htmlClass{term term-4}{\log}\,\htmlClass{term term-5}{p_\theta(y \mid x)}$$

Symbol	What it means
$p_\theta(y \mid x)$	trainedHow likely the model with weights $\theta$ makes answer $y$ for prompt $x$ (the product from step 1). For the coin, this is the “0.00222” for one knob setting.
$x$, $y$	dataOne prompt and its answer, a single row of the dataset.
$\log$	mathTurns the multiplication into addition. It never changes the order (a bigger probability always has a bigger log), so the winner stays the same.
$\mathcal{D}$“calligraphic D”	dataThe dataset: the full collection of $(x, y)$ pairs you train on.
$\sum_{(x,y)\in\mathcal{D}}$“sum over every pair x, y in D”	mathAdd the term up over every example in the dataset, just as you multiplied over every toss. $\in$ reads “belonging to”.
$\operatorname*{arg\,max}_\theta$“arg max over theta”	math“The $\theta$ that makes this sum the biggest.” The arg means it returns the knob setting, not the biggest value: for the coin, max = 0.00222 but arg max = 0.7.
$\theta$	trainedAny candidate setting of the weights, the thing being searched over.
$\theta^\star$“theta star”	resultThose best weights: the result of training. A star on a symbol means “the best one”.

The coin, written the same way, with its log values:

$$h^\star = \operatorname*{arg\,max}_h \big[\,7\log h + 3\log(1-h)\,\big] = 0.7$$

$h$	trainedThe coin model's only knob: its chance of heads, a number between 0 and 1.
$7\log h$	data7 heads were seen, each with probability $h$: seven factors of $h$ become $7 \times \log h$ after the log.
$3\log(1-h)$	data3 tails were seen, each with probability $1-h$.
$h^\star$	resultThe knob value that makes the 10 tosses most likely: 0.7, the fraction of heads.

h = 0.5: 10 × log 0.5 = −6.93 h = 0.7: 7 × log 0.7 + 3 × log 0.3 = −2.50 − 3.61 = −6.11 ← highest (least negative) h = 0.9: 7 × log 0.9 + 3 × log 0.1 = −0.74 − 6.91 = −7.65 same ranking as before: log 0.00098 = −6.93, log 0.00222 = −6.11, log 0.00048 = −7.65

e · Why training “walks downhill”

With one knob you can try every value and pick the peak. With billions of knobs you cannot. So training:

flips the sign, so “maximise the likelihood” becomes “minimise the loss” $-\log p$ (the next step on this page);
measures which way to nudge each weight so the loss goes down: the gradient;
takes a small step and repeats, millions of times.

Each step makes the dataset's answers a little more likely. That is the Take a training step button in the lab above, done in billions of dimensions at once.

Logs turn the product into a sum

Multiplying hundreds or thousands of probabilities gives numbers so small the computer rounds them to zero. Adding their logs gives the same information as an ordinary number. Throughout this page “log” means the natural logarithm, written ln on a calculator.

log 0.30 + log 0.60 = −1.20 + (−0.51) = −1.71 and log 0.18 = −1.71 ✓ 300 tokens at p = 0.9 each: product 0.9³⁰⁰ = 0.0000000000000187 (at about 1,000 tokens it drops below the smallest float32 number and becomes 0) sum of logs 300 × (−0.105) = −31.6 (an ordinary number)

Flip the sign and average: that is the loss

Optimisers are built to push numbers down, so flip the sign. Divide by the number of answer tokens to get the average surprise per token: $1.71 / 2 = 0.86$. This is the cross-entropy loss, also called negative log-likelihood. (The lab above also grades the closing <|end|>, at $p = 0.40$: $(1.71 + 0.92)/3 = 0.88$, the average it shows before training.)

p(correct token)	loss −log p	meaning
0.99	0.01	almost certain and right
0.50	0.69	a coin flip
0.10	2.30	quite wrong
0.01	4.61	very wrong
0.0001	9.21	completely surprised

Perplexity $= e^{\mathcal{L}}$ says the same thing in “number of choices”: a loss of 0.69 is a perplexity of 2, as unsure as picking between two equally likely tokens.

What one training step does: the gradient

The model turns raw scores $z$ (logits) into probabilities with softmax. The slope of the loss with respect to each score has a very simple form:

$$\frac{\partial \mathcal{L}}{\partial z_i} = p_i - \mathbb{1}[i = y]$$

$z_i$“z sub i”, a logit	trainedThe raw score the model gives to token $i$ of the vocabulary, before it becomes a probability. Any real number, positive or negative.
$i$	mathAny token of the vocabulary: the formula holds for each one separately.
$p_i$	trainedThe probability of token $i$ after softmax: $p_i = e^{z_i} / \sum_j e^{z_j}$. Softmax makes every score positive and makes them add up to 1.
$y$	dataHere: the correct token at this position, from the training example.
$\frac{\partial \mathcal{L}}{\partial z_i}$“partial L over partial z-i”	resultThe slope: how much the loss changes if you nudge score $z_i$ up a little. Training moves each score against its slope, so a positive slope means “push this score down”.
$\mathbb{1}[i = y]$“indicator of i equals y”	mathAn on/off switch: 1 when token $i$ is the correct one, 0 for every other token.

The correct token $y$ is pushed up by $1 - p_y$. Every other token is pushed down by its own probability. A token already at 0.99 gets a push of only 0.01: there is little left to learn.

before: " Paris" 0.30 " Lyon" 0.25 " The" 0.10 all others 0.35 push: +0.70 −0.25 −0.10 −0.35 (up and down balance: 0.70 = 0.25 + 0.10 + 0.35)

SFT never pushes down a specific bad answer on purpose. It only pushes up the shown answer; everything else loses probability as a side effect. Preference methods (the next family) add the missing “push this one down”.

SFT copies proportions, and that is W1

Minimising cross-entropy over a dataset is the same as making the model's distribution as close as possible to the data's distribution, measured by KL divergence:

$$\mathbb{E}_{y\sim p_{\text{data}}}\big[-\log p_\theta(y\mid x)\big] = \underbrace{H(p_{\text{data}})}_{\text{fixed floor}} + \underbrace{\mathrm{KL}\big(p_{\text{data}}\,\|\,p_\theta\big)}_{\text{what training shrinks}}$$

$p_{\text{data}}$“p data”	dataHow often each answer appears in your dataset for this prompt. If the expert answers A 70% of the time and B 30%, then $p_{\text{data}}(A) = 0.7$, $p_{\text{data}}(B) = 0.3$.
$y \sim p_{\text{data}}$“y drawn from p data”	mathThe tilde $\sim$ means “is sampled from”: $y$ is an answer picked from the dataset, in the dataset's proportions.
$\mathbb{E}_{y\sim p_{\text{data}}}[\,\cdot\,]$“expected value over y”	mathThe average of the bracket over those draws, each answer weighted by how often it appears. With A 70% and B 30%: $0.7 \times \text{(loss on A)} + 0.3 \times \text{(loss on B)}$.
$-\log p_\theta(y\mid x)$	trainedThe SFT loss on one answer, so the left side is the SFT loss averaged over the dataset.
$H(p_{\text{data}})$“entropy of p data”	dataEntropy, $H(p) = -\sum_y p(y)\log p(y)$: how varied the data itself is. It depends only on the data, so no choice of weights can lower it. It is the floor of the loss.
$\mathrm{KL}\big(p_{\text{data}} \,\\|\, p_\theta\big)$“KL divergence from p data to p theta”	resultKL divergence, $\sum_y p_{\text{data}}(y)\log\frac{p_{\text{data}}(y)}{p_\theta(y\mid x)}$: a distance-like gap between the data's proportions and the model's. It is 0 only when the two match exactly, and positive otherwise. The double bar $\\|$ just separates the two distributions; their order matters.
$\underbrace{\;\cdots\;}_{\text{note}}$	mathAn underbrace is a label, not maths: it names the part above it.

So the best possible model copies the data's proportions exactly, bad habits included. Try it: the expert picks option A some of the time and B the rest. Move the model's probability and find the lowest loss.

Where is the lowest loss?

The curve is the average loss on the expert's choices for every possible model. Its lowest point always sits at the expert's own split.

9

Watch out for

It copies bad habits W1
SFT has no idea which examples were good. Whatever is in the data, in whatever proportion, is what it learns.
Mistakes snowball W2
During training the model always sees the correct previous tokens (teacher forcing). At use time it reads its own output, mistakes included, a situation it never practised. This gap is called exposure bias.
The ceiling is the teacher W3
Imitation aims to equal the examples. Nothing in the loss rewards finding a better answer than the ones it was shown; small gains sometimes come from averaging out the examples' noise, not from the objective.
Silent truncation
If a row is longer than the maximum length, its end is cut off, and the end is usually the answer. The model then learns from half an answer. Check lengths before training: TRL's SFTConfig cuts at max_length = 1024 tokens by default.
Forgetting to mask the prompt
Without the mask most of the signal goes into predicting the prompt. Try the toggle in the lab: the answer's share of the signal drops sharply.
Memorising a small dataset
With a few thousand rows and many passes, the model memorises. Watch the validation loss: if it rises while the training loss keeps falling, stop.

Keep only the good rows

Filtered SFT Rejection-sampling fine-tuning (RFT)

The same training as plain SFT, but first a checker, a judge or a person removes the bad answers. The model can only copy what survives the filter.

Watch the judge's switch remove a row from the loss, and the kept answers share the target.

Changes vs plain SFT: Which rows you keep
Fixes: W1 copies the bad with the good
Cost: ●●●●● the cheapest first fix
Library: Build it: filter with your scores, then SFTTrainer

1

The idea in plain words

Plain SFT copies everything it sees. The simplest fix is to stop showing it the bad examples. Score every (prompt, answer) row, keep the rows that clear a bar, and train on those exactly as in #1.

It goes by two names. When you filter data you already have, people say filtered SFT. When you generate the data on purpose (sample many answers from a model, reject the ones that fail a check, keep the rest), it is called rejection-sampling fine-tuning.

Run that loop again with the improved model and you get expert iteration, the idea behind STaR and ReST, a later technique in the catalogue.

2

See it work

Filter ten sampled answers

A model answered the same question ten times. Pick who does the filtering and watch what the model would learn. Then reveal the true quality to see what each filter got wrong.

3

Where the training signal comes from

Two decisions: where the candidate answers come from, and who decides which ones to keep.

4

A small example

Kept: the checker passes it, the judge gives 9

User · promptA shop sells 3 pens for $2. How much do 12 pens cost?

Sampled answer · kept12 ÷ 3 = 4 bundles. 4 × $2 = $8.

Rejected: wrong total, the judge gives 2

User · promptA shop sells 3 pens for $2. How much do 12 pens cost?

Sampled answer · dropped3 pens cost $2, so 12 pens cost 12 × $2 = $24.

The judge's own exchange

To the judge · rubricScore the answer from 1 to 10. 10 = correct result with clear, correct working. Reply as: score: N, then one sentence.

Judgescore: 9. Correct method (group into bundles of 3) and correct total, shown clearly.

5

What a training batch looks like

Prompt	Candidate answer	Checker	Judge	Keep?
3 pens cost $2. Price of 12?	12 ÷ 3 = 4 bundles. 4 × $2 = $8.	✓ $8	9	keep
3 pens cost $2. Price of 12?	3 pens cost $2, so 12 pens cost 12 × $2 = $24.	✗ $24	2	drop
Capital of Australia?	The capital of Australia is Canberra.	✓	9	keep
Capital of Australia?	Sydney is the capital of Australia.	✗	1	drop
Summarise: “The meeting moved from Monday to Wednesday because the room was booked.”	The meeting is now on Wednesday because the room was taken on Monday.	n/a	8	keep
Summarise: “The meeting moved from Monday to Wednesday because the room was booked.”	There was a meeting.	n/a	3	drop

Prompt	Target answer
A shop sells 3 pens for $2. How much do 12 pens cost?	12 ÷ 3 = 4 bundles. 4 × $2 = $8.
What is the capital of Australia?	The capital of Australia is Canberra.
Summarise: “The meeting moved from Monday to Wednesday because the room was booked.”	The meeting is now on Wednesday because the room was taken on Monday.

# candidates.jsonl: scores are kept beside each row while you filter
{"prompt": "A shop sells 3 pens for $2. How much do 12 pens cost?", "completion": "12 ÷ 3 = 4 bundles. 4 × $2 = $8.", "checker": true, "judge": 9}
{"prompt": "A shop sells 3 pens for $2. How much do 12 pens cost?", "completion": "3 pens cost $2, so 12 pens cost 12 × $2 = $24.", "checker": false, "judge": 2}

# train.jsonl: what SFTTrainer receives. The score columns are gone.
{"prompt": "A shop sells 3 pens for $2. How much do 12 pens cost?", "completion": "12 ÷ 3 = 4 bundles. 4 × $2 = $8."}

6

The loss

$$\htmlClass{term term-1}{\mathcal{L}_{\text{filt}}(\theta)} \;=\; -\,\htmlClass{term term-2}{\frac{1}{N_{\text{keep}}}}\sum_{(x,y)\in\mathcal{D}} \htmlClass{term term-3}{\mathbb{1}\!\big[\,\htmlClass{term term-4}{r(x,y)}\ge \htmlClass{term term-5}{\kappa}\,\big]}\;\htmlClass{term term-6}{\log p_\theta(y\mid x)}$$

Read it aloud: “It is the plain SFT loss, but every row is multiplied by a switch. The switch is 1 for rows whose score reaches the bar and 0 for the rest, so dropped rows contribute nothing.” Point at a row of the table to light its term up in the formula.

whole set This formula is written over the whole candidate set $\mathcal{D}$. The loss of one row is the term inside the sum: the switch times that row's SFT loss, $\mathbb{1}[r \ge \kappa]\cdot\big(-\log p_\theta(y\mid x)\big)$. The $\frac{1}{N_{\text{keep}}}$ in front turns the sum into an average over the kept rows. Station 7 follows one full training step, from the batch to the update.

Every symbol in the formula

Symbol	What it means
$\mathcal{L}_{\text{filt}}(\theta)$“L-filt of theta”	resultThe filtered SFT loss: one number for the whole candidate set, lower is better.
$\theta$“theta”	trainedThe model's weights, the only thing training changes.
$\mathcal{D}$“calligraphic D”	dataAll the candidate rows, before filtering: the six rows of the “Before filtering” tab, good and bad together.
$(x, y)$	dataOne candidate row: a prompt $x$ and an answer $y$. The answer may have been written by people, by another model or by the model itself.
$\sum_{(x,y)\in\mathcal{D}}$“sum over every row in D”	mathGo through every candidate row and add up the term on the right. $\in$ reads “belonging to”.
$r(x,y)$“r of x, y”, the score	frozenThe score of answer $y$ to prompt $x$: from a checker (0 or 1), a judge model (1–10) or a person. It is computed once, before training, and never changes.
$\kappa$“kappa”, the bar	settingThe pass mark you choose, for example 7 on a 1–10 judge. Raise it and fewer, cleaner rows survive.
$\ge$	math“Is at least”: $r(x,y) \ge \kappa$ is true when the score reaches the bar.
$\mathbb{1}[\,\cdot\,]$“indicator of …”, the keep switch	mathEquals 1 if the condition inside the brackets is true and 0 if not. Multiplying a row's loss by 0 deletes it: a dropped row is as if it never existed.
$\frac{1}{N_{\text{keep}}}$“one over N-keep”	mathAverage over the kept rows only. $N_{\text{keep}} = \sum \mathbb{1}[r \ge \kappa]$ counts the rows that passed, so dropped rows do not count in the average either.
$\log p_\theta(y\mid x)$	trainedThe model's log-probability of the whole answer $y$ given $x$: $\sum_t \log p_\theta(y_t\mid x, y_{<t})$, one term per answer token. In practice it is averaged per token, exactly as in #1.
$-$	mathFlips the negative log-probability into a positive loss to minimise.

What you must supply: candidate rows $(x, y)$ and a way to score them, $r$. You choose the bar $\kappa$. After filtering, the trainer sees ordinary SFT rows: the score never reaches the loss as a number, only as keep or drop.

How the loss works, step by step

Score every candidate. The checker, judge or person gives each row a score $r(x,y)$.
Flip the switch. Compare each score with the bar $\kappa$: the switch $\mathbb{1}[r \ge \kappa]$ is 1 (keep) or 0 (drop).
Compute the #1 loss on each row: the per-token surprise $-\log p_\theta$ of its answer.
Multiply by the switch and average over the kept rows. Dropped rows add 0 and are not counted.
Step downhill. The model raises the probability of the kept answers only.

Worked example: the six candidate rows of the batch above, bar $\kappa = 7$ (per-token losses are illustrative)

row judge r r ≥ 7 ? switch −log p per token counts 1 $8 (correct) 9 yes 1 0.40 0.40 2 $24 (wrong) 2 no 0 1.10 0 3 Canberra 9 yes 1 0.20 0.20 4 Sydney (wrong) 1 no 0 0.60 0 5 good summary 8 yes 1 0.90 0.90 6 vague summary 3 no 0 0.30 0 N_keep = 3 L_filt = (0.40 + 0.20 + 0.90) / 3 = 0.50

Look at row 4: the model already finds “Sydney” fairly likely, and filtering does nothing about it. A dropped row is ignored, not punished. Filtering can only stop the model learning a mistake from the data; it cannot push down a mistake the model already makes. Pushing down needs the preference family.

7

One training step on the GPU

Training is the plain SFT step of #1, on the rows that passed the bar. The new work, writing and judging the candidates, happens before training starts. Plain SFT's station 7 walks through the full step with every number; this one shows what changes.

1BatchB kept rows; dropped rows never reach the GPU

2Forwardas #1: every position of every row at once

3Loss#1's loss, averaged over the kept rows

4Backwardp − y at each answer token, as #1

5Updateonce; only kept answers pull the weights

One step, slope by slope

Plain SFT's step on the rows that passed the bar. Take a batch of 3 kept rows whose answers have 10 tokens each.

3 kept rows, 10 answer tokens each → ONE training step 0 before candidates were written and judged; rows under κ were deleted and never reach the GPU 1 forward one pass over the 3 kept rows: every position of every row at once 2 loss #1's loss on each row, averaged over the 3 rows 3 backward 30 slopes at the same time, one per answer token of each kept row: g_t = k (p_t − y_t) k = 1/(3 × 10) = 1/30 each one from its own p and its own target only 4 add each weight sums the 30 contributions → 1 gradient per weight 5 update every weight moves once

The judge's score is in no slope. It only decided which rows are in the batch: a kept row scored 7 and one scored 10 get the same k.
A dropped row makes no slope at all. It is ignored, not pushed down.
Not one after the other. The 30 slopes are measured on the same old weights and added into each weight; the weights move once, as in #1.
Read, not graded: the prompt tokens, as in #1.

1The batch

Before training, a model writes several candidate answers per prompt and the judge scores them. Rows under $\kappa$ are deleted. The batches are then filled with kept rows only, so the switch $\mathbb{1}[r \ge \kappa]$ never appears in the training code.

2Forward pass

Exactly #1's: the kept rows are padded into one grid and every position of every row runs every layer at once.

3Loss: from each row to one number

Each row gets #1's loss; the batch averages them. Read as a formula, dropped rows would sit in the sum with a switch of 0; in practice they are simply not in the batch.

as the formula reads it, the six candidates in one batch: (0.40 + 0.20 + 0.90) / 3 = 0.50 3 rows count, 3 count zero as trainers do it, filtered before batching: a batch of 6 kept rows (last three illustrative) (0.40 + 0.20 + 0.90 + 0.30 + 0.60 + 0.70) / 6 = 0.52

4Backward pass: the slope at every answer token

($y_t$ is 1 at the target token and 0 at every other word.) Exactly #1's: each answer token of row i gets $g_t = \frac{1}{B\,\sum m_i}(p_t - y_t)$, and the chain rule carries it down. A dropped row sends nothing because it is not in the batch: it is ignored, not pushed down. That is the difference from the preference family, where a bad answer gets a slope of its own, pointing down.

5Update

Every weight moves once, by the sum of the kept rows' pushes.

On a real GPU

Generating is the slow part. A training step reads a whole answer in one pass, but writing it takes one forward pass per token. Trainers generate with a dedicated inference engine, often on other machines.
Judging costs at least one forward pass of a judge model per candidate (many more if the judge writes its reasoning before the score), or one run of a checker (tests, a calculator).
The training step itself costs what #1's costs, per kept row.

8

The maths behind it

Filtering edits the distribution the model is forced to copy

From #1: SFT makes the model match the data's proportions. Filtering changes those proportions before training starts. The new target is the old one with the failing answers removed and the rest scaled back up to 100%:

$$p_{\text{keep}}(y\mid x) = \frac{p_{\text{data}}(y\mid x)\;\mathbb{1}[r(x,y)\ge\kappa]}{\sum_{y'} p_{\text{data}}(y'\mid x)\;\mathbb{1}[r(x,y')\ge\kappa]}$$

$p_{\text{data}}(y\mid x)$	dataHow often answer $y$ appears for prompt $x$ in the original candidates (A 70%, B 30%).
$\mathbb{1}[r(x,y)\ge\kappa]$	mathThe keep switch: 1 if the answer's score $r$ reaches the bar $\kappa$, else 0.
top line	mathThe original share of $y$ if it is kept, 0 if it is dropped.
$\sum_{y'}$“sum over y-prime”	mathAdd up over every possible answer. The prime in $y'$ only gives the counting variable a different name from the $y$ on top.
bottom line	mathThe total share that survives the filter (70% here). Dividing by it rescales the survivors so they add up to 100% again.
$p_{\text{keep}}(y\mid x)$	resultThe new distribution the model will copy: the old one with failing answers removed and the rest scaled up.

data: method A (good) 70% method B (bad) 30% filter: drop every B target: method A 70 / 70 = 100% method B 0% → the model learns ≈ 100 / 0 instead of 70 / 30

A real judge makes mistakes: how much bad slips through?

Call $b$ the share of bad rows in the data, $g$ the chance the judge keeps a good row (its recall) and $f$ the chance it wrongly keeps a bad one (its false-positive rate). Among the kept rows, the bad share is:

$$\text{bad share after filtering} = \frac{b\,f}{(1-b)\,g + b\,f}$$

$b$	dataThe share of bad rows in the candidates, before filtering (0.30 = 30%).
$1-b$	dataThe share of good rows (0.70).
$g$	frozenThe judge's recall: the chance it keeps a good row (0.90 keeps 9 good rows in 10).
$f$	frozenThe judge's false-positive rate: the chance it wrongly keeps a bad row (0.10).
$(1-b)\,g$	mathThe share of all candidates that are good and kept. Two letters side by side are multiplied.
$b\,f$	mathThe share of all candidates that are bad and kept: the leak.
the fraction	resultBad-and-kept divided by everything kept: the share of bad rows in what the model will copy.

b = 0.30, g = 0.90, f = 0.10 kept good = 0.70 × 0.90 = 0.63 kept bad = 0.30 × 0.10 = 0.03 bad share = 0.03 / (0.63 + 0.03) = 4.5% ← down from 30%, but not zero

Judge quality calculator

Change the data and the judge. The bar shows what the model will copy.

Rejection sampling: training on your own successes

When the candidates come from the model itself, keeping the ones that pass means sampling from the model's distribution restricted to successes. Training on them shifts the model toward its own best behaviour.

$$p(y \mid x, \text{pass}) \;\propto\; p_\theta(y\mid x)\;\mathbb{1}[\text{pass}(x,y)]$$

$p_\theta(y\mid x)$	trainedHow likely the current model is to write answer $y$ for prompt $x$: how often that answer shows up when you sample it.
$\text{pass}(x,y)$	frozenThe checker's verdict on answer $y$: true if it passes.
$\mathbb{1}[\text{pass}(x,y)]$	math1 for passing answers, 0 for failing ones: the filter.
$\propto$“is proportional to”	mathEqual up to a constant scale. The right side does not add up to 1; dividing by its total (as in step 1) fixes that, and the shape is what matters.
$p(y \mid x, \text{pass})$	resultThe distribution of the answers you keep: the model's own answers, given that they passed.

the model solves a question 1 time in 4 → p(fail) = 0.75 sample 8 answers: p(at least one passes) = 1 − 0.75⁸ = 1 − 0.10 = 0.90 → 90% of such questions yield a training example the model can learn from

Why it cannot beat the grader

The target is now “whatever passes $r$”. If the score is wrong, the target is wrong in the same way. Filtering moves the ceiling from the data to the grader; it does not remove it (W3).

9

Watch out for

The grader becomes the ceiling
A bad row marked good teaches a mistake; a good row marked bad throws away a lesson. Check the judge against human labels on a sample before trusting it.
Fewer rows, more memorising
Filtering shrinks the dataset. With few rows left, the model overfits faster. Watch the validation loss.
Judge steps, not whole runs
In multi-step tasks, “keep only successful runs” throws away failed runs that still contain many good steps, and the off-track situations #6 needs.
It teaches the absence of bad
The model never sees a tempting wrong answer next to the reason to avoid it. That contrast needs #6, #8 or the preference family.
Easy prompts take over
In rejection sampling, a question the model solves 8 times out of 8 yields 8 kept rows; a hard one yields 0 or 1. Cap or deduplicate the kept answers per prompt, or the set drifts toward what the model already knows.
Checker loopholes
A checker that only reads the final number keeps “$0.75 × 12 = $8”: wrong working, lucky result. Combine it with a judge when the working matters.

A stronger model writes the answers

Distillation Teacher → student · knowledge distillation

A large, strong model (the teacher) produces the answers, and a smaller model (the student) learns to reproduce them. With access to the teacher's probabilities, the student can copy what the teacher said and also how sure it was about every alternative.

Compare the teacher's soft targets with its text alone, on the same tiny transformer.

Changes vs plain SFT: Who writes the answers, and optionally a softer target
Fixes: Lifts the ceiling to the teacher's level W3 raised, not removed, and fills gaps in coverage
Cost: ●●●●● teacher calls + plain SFT
Library: Ready: text → SFTTrainer. Probabilities → DistillationTrainer or GOLDTrainer (exp.), both on-policy by default; the offline recipe below needs a custom loss

1

The idea in plain words

There are two very different ways to distil.

Sequence-level (black box). Send prompts to the teacher, keep the text it writes, and run plain SFT on (prompt, teacher answer). This works with any teacher, including one you can only reach through an API. From the student's side, it is just #1 with a better writer.

Logit-level (white box). With an open-weights teacher you can read its full probability distribution at every token. The student is trained to match that whole distribution, not only the one token the teacher picked. The near-misses (“sofa” is plausible here, “banana” is not) are called dark knowledge, and they make each example worth much more.

Distillation's other strength is coverage: you can ask the teacher about situations your recorded data never visited.

2

See it work

Hard target vs soft target

The teacher knows “mat” is most likely, “sofa” and “floor” are reasonable, and “banana” is absurd. Train the student with each kind of target and compare what it learns about the alternatives.

3

Where the training signal comes from

Three choices: the teacher, the prompts it answers, and whether anyone checks it.

4

A small example

The teacher answers

User · promptExplain in one sentence why ice floats on water.

Teacher · large modelIce is less dense than liquid water, because freezing locks the molecules into an open hexagonal pattern that takes up more space.

The student, before and after

Student · beforeIce floats because it is cold and light.

Student · after distillationIce floats because it is less dense than water: freezing arranges the molecules into an open structure that takes up more space.

What the logit view adds, for the first token

The teacher wrote Ice, but its distribution at that spot was:

"Ice" 0.81 "Because" 0.09 "Water" 0.06 "When" 0.04

A hard target says only “Ice”. A soft target also says that starting with “Because” would have been fine.

5

What a training batch looks like

Prompt	Teacher's answer = target	Source
Explain in one sentence why ice floats on water.	Ice is less dense than liquid water, because freezing locks the molecules into an open hexagonal pattern that takes up more space.	teacher-large
Give a synonym for “happy”.	Cheerful.	teacher-large
Is 91 a prime number?	No. 91 = 7 × 13.	teacher-large
Write a title for a blog post about saving water at home.	Ten Small Habits That Cut Your Water Bill	teacher-large

Position	Token written	Teacher's top probabilities (the soft target)
1	No	No 0.92 · The 0.04 · 91 0.03 · Yes 0.01
2	.	. 0.71 · , 0.27 · ! 0.02
3	␣91	␣91 0.66 · ␣It 0.30 · ␣Because 0.04
4	␣=	␣= 0.58 · ␣is 0.40 · ␣equals 0.02
5	␣7	␣7 0.97 · ␣13 0.03

# sequence-level: ordinary SFT rows, the teacher wrote the completion
{"prompt": "Is 91 a prime number?", "completion": "No. 91 = 7 × 13."}

# logit-level, for a custom offline trainer: the teacher's top-k log-probabilities for every answer token
{"prompt": "Is 91 a prime number?", "completion": "No. 91 = 7 × 13.",
 "teacher_topk": [{"No": -0.08, "The": -3.2, "91": -3.5}, {".": -0.34, ",": -1.31}, ...]}

6

The loss

Sequence-level: plain SFT on the teacher's text

$$\mathcal{L}_{\text{seq}}(\theta) = -\,\mathbb{E}_{x}\;\mathbb{E}_{\htmlClass{term term-8}{y\sim p_{\text{tea}}(\cdot\mid x)}}\big[\log p_\theta(y\mid x)\big]$$

Read it aloud: “Over many prompts $x$, let the teacher write an answer $y$. Then make that answer as likely as possible for the student.” This is the plain SFT loss of #1; the only new thing is who wrote the target.

Logit-level: match the teacher's whole distribution

$$\htmlClass{term term-1}{\mathcal{L}_{\text{KD}}(\theta)} = \htmlClass{term term-2}{\alpha}\;\htmlClass{term term-3}{T^{2}}\,\frac{1}{|y|}\sum_{t=1}^{|y|}\htmlClass{term term-4}{\mathrm{KL}}\Big(\htmlClass{term term-5}{\operatorname{softmax}\!\big(z^{\text{tea}}_t/T\big)}\;\Big\|\;\htmlClass{term term-6}{\operatorname{softmax}\!\big(z^{\theta}_t/T\big)}\Big) \;+\; \htmlClass{term term-7}{(1-\alpha)\,\mathcal{L}_{\text{SFT}}(\theta)}$$

Read it aloud: “At every answer position, soften both the teacher's and the student's distributions with temperature $T$, measure how far apart they are with KL, and average over the answer. Blend that with the ordinary SFT loss on the token actually written.” Point at a row of the table to light its term up in the formulas.

one row Both losses are for one row: one prompt with its answer. In $\mathcal{L}_{\text{seq}}$, $\mathbb{E}_x$ means “averaged over the rows”. $\mathcal{L}_{\text{KD}}$ is written for one row directly: its $\frac{1}{|y|}\sum_t$ runs over the positions of that row's answer. Station 7 follows one full training step, from the batch to the update.

Every symbol in the two formulas

Symbol	What it means
Shared by both formulas
$\theta$“theta”	trainedThe student's weights, the only thing training changes.
$x$	dataA prompt, for example “Is 91 a prime number?”.
$y$	frozenThe answer the teacher wrote for $x$: the target text. It is $y_1, y_2, \dots$, one token per position.
$p_{\text{tea}}$“p teacher”	frozenThe teacher: a larger or better model whose weights never change.
$p_\theta$“p theta”	trainedThe student: the smaller model being trained.
Sequence-level formula
$\mathcal{L}_{\text{seq}}(\theta)$	resultThe sequence-level distillation loss: plain SFT where the targets are the teacher's answers.
$y\sim p_{\text{tea}}(\cdot\mid x)$“y drawn from the teacher, given x”	frozenThe answer is sampled from the teacher: the teacher writes it, token by token. The dot $\cdot$ stands for “every possible answer”: $p_{\text{tea}}(\cdot\mid x)$ is the teacher's whole spread of possible answers to $x$.
$\mathbb{E}_x$, $\mathbb{E}_{y\sim\dots}$“expected value”	mathAverages: over the prompts in your dataset, and over the answers the teacher might write. In practice: one teacher answer per prompt, averaged over the batch.
$\log p_\theta(y\mid x)$	trainedThe student's log-probability of the teacher's whole answer, the sum of one log-probability per token (#1). The leading minus sign makes the loss positive.
Logit-level formula
$\mathcal{L}_{\text{KD}}(\theta)$“L-KD”, knowledge distillation	resultThe logit-level distillation loss, averaged over the $\|y\|$ answer positions like $\mathcal{L}_{\text{SFT}}$, so the two parts are on the same scale.
$t$, $\|y\|$	mathA position in the answer, and the number of answer tokens. $\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}$ is “average over the answer positions”.
$z^{\text{tea}}_t$“z teacher at t”	frozenThe teacher's logits at position $t$: one raw score per vocabulary token, before they become probabilities. You need access to the teacher's model, or to its saved top-k log-probabilities, to have them.
$z^{\theta}_t$“z theta at t”	trainedThe student's logits at the same position, computed on the same history: the teacher's answer so far.
$T$“temperature”	settingA number you choose, usually 1 to 4. Dividing every logit by $T > 1$ flattens the distribution so the small probabilities of the runner-up tokens stay visible. $T = 1$ changes nothing.
$\operatorname{softmax}\!\big(z^{\text{tea}}_t/T\big)$	frozenThe teacher's softened probability list, the soft target: $\operatorname{softmax}(z)_i = e^{z_i}/\sum_j e^{z_j}$ turns any scores into positive numbers that add up to 1.
$\operatorname{softmax}\!\big(z^{\theta}_t/T\big)$	trainedThe student's probabilities, softened the same way.
$\mathrm{KL}(q\,\\|\,p)$“KL divergence of q from p”	math$\sum_i q_i \log(q_i/p_i)$, summed over every vocabulary token $i$: how far the student's list $p$ is from the teacher's $q$. It is 0 only when they are identical. The double bar $\\|$ separates the target (left) from the list being trained (right).
$T^{2}$“T squared”	settingSoftening shrinks the gradient by about $1/T^2$ (step 4 below). Multiplying by $T^2$ keeps the soft loss as strong as the hard one.
$\alpha$“alpha”	settingHow much weight goes to the soft target versus the hard one, between 0 and 1. 0.5 is a common start; $\alpha = 1$ uses only the teacher's distribution.
$(1-\alpha)\,\mathcal{L}_{\text{SFT}}(\theta)$	trainedThe ordinary SFT loss of #1 on the token actually written (the hard label), with the remaining weight $1-\alpha$.

What you must supply: prompts $x$. The teacher writes the answers. For the logit-level loss you also need the teacher's probabilities at every answer position, so the teacher must be a model you can run (or one whose API returns log-probabilities); you choose $T$ and $\alpha$.

How the losses work, step by step

Sequence-level $\mathcal{L}_{\text{seq}}$

The teacher answers every prompt once, before training.
Each prompt and teacher answer becomes an ordinary SFT row.
Train with the #1 loss. Per position the student only learns “the teacher wrote this token”.

Logit-level $\mathcal{L}_{\text{KD}}$

Feed the prompt and the teacher's answer to both models. At every answer position each returns its logits.
Divide both sets of logits by $T$ and apply softmax: two softened probability lists.
Measure their gap with KL, multiply by $T^2$ and $\alpha$.
Add $(1-\alpha)$ times the ordinary SFT loss on the token the teacher actually wrote.
Average over the answer positions and step downhill. The student moves toward the teacher's whole opinion, including how plausible the alternatives were.

Worked example: position 1 of “Is 91 a prime number?”, three tokens only, $T = 2$, $\alpha = 0.5$

"No" "The" "91" teacher logits z_tea 2.0 0.7 0.0 student logits z_θ 0.9 0.6 0.3 softened, T = 2: teacher q 0.53 0.28 0.19 student p 0.38 0.33 0.29 KL(q‖p) = 0.169 − 0.050 − 0.074 = 0.045 soft part α · T² · KL = 0.5 × 4 × 0.045 = 0.090 hard part (1 − α) · (−log p_θ("No")) at T = 1: student gives "No" 0.44 = 0.5 × 0.83 = 0.414 L_KD at this position = 0.090 + 0.414 = 0.50

The hard part pushes “No” up and every other token down. The soft part says how far down: “The” should stay above “91”, because the teacher thought it was a fair second choice. With $\alpha = 1$ only the soft part remains.

7

One training step on the GPU

The sequence-level loss is plain SFT on the teacher's text, so its step is #1's. The logit-level loss adds a second model to every step: the teacher reads the same batch, frozen, and its probabilities become the student's targets. Plain SFT's station 7 walks through the full step with every number; this one shows what changes.

1BatchB rows: prompt + the teacher's answer

2Forwardstudent with gradient; teacher frozen, no gradient

3Losssoft KL + hard SFT at each answer token, averaged

4BackwardαT(p̃ − q̃) + (1 − α)(p − y) at each token; student only

5Updatethe student moves; the teacher never does

One step, slope by slope

Sequence-level distillation is #1's step on the teacher's text: 10 answer tokens make 10 slopes with k = 1/10. The logit-level step adds the teacher's lists. Take one row whose teacher answer has 10 tokens.

one row: prompt + the teacher's 10-token answer → ONE training step (logit-level) 1 forward student: one pass over the row, every position at once, with gradient teacher: one pass over the same row, frozen, no gradient → its list q at each of the 10 positions 2 loss per position: α T² KL(q̃ ‖ p̃) + (1 − α)(−log p(target)), averaged over the 10 3 backward 10 slopes at the same time, one per answer position: g_t = (1/10) [ αT (p̃_t − q̃_t) + (1 − α)(p_t − y_t) ] each one from its own position only: the student's list and the teacher's list there 4 add each weight of the student sums the 10 contributions → 1 gradient per weight 5 update the student moves once; the teacher never does

No single k. The hard part is #1's p − y with weight 1 − α. The soft part gives every word its own slope, toward the teacher's softened list. Both are divided by 10 here, and also by B in a batch of B rows.
The teacher's lists come first. The slope at position t needs the teacher's list at that same position, so the teacher's forward pass (or its stored top-k lists) must be ready before the backward pass. It needs nothing from the other 9 positions.
Not one after the other. The 10 slopes are measured on the same old weights and added into each weight; the weights move once, as in #1.

1The batch

Each row is a prompt with the teacher's answer, padded into one grid of B rows as in #1. The same grid goes to both models.

2Forward pass

The student runs #1's forward pass and keeps its activations for the backward pass. The teacher runs the same grid through its own network, usually a bigger one, frozen and with no gradient: it only hands over its logits at every answer position. For the soft part both lists are softened with the same temperature T; the hard part uses the student's ordinary probabilities.

3Loss: from each row to one number

Each row's loss is the soft part plus the hard part, averaged over its answer positions; the batch averages the rows (or pools the answer tokens, as in #1).

row L_KD of the row, averaged over its answer positions (illustrative) 1 0.50 2 0.62 3 0.35 4 0.41 batch loss = (0.50 + 0.62 + 0.35 + 0.41) / 4 = 0.47

4Backward pass: the slope at every answer token

$$g_t = \frac{1}{B\,|y|}\Big[\,\alpha\,T\,\big(\tilde p_t - \tilde q_t\big) \;+\; (1-\alpha)\,\big(p_t - y_t\big)\Big]$$

$\tilde p$ and $\tilde q$ are the student's and the teacher's lists softened with the temperature T, and $y_t$ is 1 at the target token and 0 at every other word. The hard part is #1's p − y. The soft part gives every word a slope toward the teacher's softened probability: words the student over-rates go down, words it under-rates go up. From the logits down, the student's backward pass is #1's; the teacher has none.

position 1 of the worked example above, before dividing by B × |y| (T = 2, α = 0.5) word soft αT(p̃ − q̃) hard (1 − α)(p − y) slope "No" −0.145 −0.282 −0.426 raised "The" +0.055 +0.162 +0.216 lowered "91" +0.090 +0.120 +0.210 lowered the soft part pushes "91" down harder than "The": the teacher found "The" a fair second choice; in the total the hard part still lowers "The" a little more, because the student over-rates it

5Update

Only the student's weights move. The teacher is the same at every step.

On a real GPU

One extra forward pass per batch. The teacher keeps nothing for a backward pass, so its pass costs about a third of a training step of the same size; but teachers are usually several times bigger than the student.
Memory holds both models: the student with gradients and optimiser state, the teacher's weights only.
Or store the teacher's lists. Run the teacher once before training and save its probabilities. A full list is 128,000 numbers per position, far too much, so trainers keep the top 20 or so per position.
Same vocabulary. Logit-level distillation needs both models to use the same tokens: a slope on word 5,000 only means something if word 5,000 is the same token for both.

8

The maths behind it

Temperature softens a distribution

$$\operatorname{softmax}(z/T)_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}$$

$z$	trainedThe logits: one raw score per token of the vocabulary, any real number.
$z_i$, $i$	mathThe score of one particular token $i$.
$T$	settingThe temperature. Dividing by $T > 1$ shrinks the gaps between scores; $T = 1$ leaves them as they are.
$e^{z_i/T}$“e to the z-i over T”	mathThe exponential, with $e \approx 2.718$. It turns any score into a positive number and keeps the order: a higher score stays higher.
$\sum_j e^{z_j/T}$	mathThe same thing added up over every token $j$. Dividing by this total makes the results add up to 1.
$\operatorname{softmax}(z/T)_i$	resultThe probability of token $i$ at temperature $T$.

logits z = [2.0, 0.7, 0.0] T = 1: e^2.0 = 7.39 e^0.7 = 2.01 e^0 = 1.00 sum 10.40 → [0.71, 0.19, 0.10] T = 2: e^1.0 = 2.72 e^0.35 = 1.42 e^0 = 1.00 sum 5.14 → [0.53, 0.28, 0.19] same order, flatter shape: the runner-up tokens become easier to learn from

KL divergence, by hand

teacher q = [0.70, 0.20, 0.10] student p = [0.50, 0.30, 0.20] KL(q ‖ p) = 0.70 · ln(0.70/0.50) + 0.20 · ln(0.20/0.30) + 0.10 · ln(0.10/0.20) = 0.70 · 0.336 + 0.20 · (−0.405) + 0.10 · (−0.693) = 0.2355 − 0.0811 − 0.0693 = 0.085
What one example teaches: hard versus soft gradients

For the same student, compare the push each target gives the three scores (from #1: gradient = prediction − target).

token A token B token C student p 0.50 0.30 0.20 hard target 1 0 0 push (target − p) +0.50 −0.30 −0.20 → keeps pushing B and C toward 0 soft target 0.70 0.20 0.10 push (target − p) +0.20 −0.10 −0.10 → stops when B = 0.20 and C = 0.10

Hard targets erase the alternatives over time. Soft targets keep the teacher's opinion of them.
Why the $T^2$ factor

The gradient of the softened KL with respect to the student's scores is $\frac{1}{T}(p^{\theta}_T - q^{\text{tea}}_T)$, where $p^{\theta}_T$ and $q^{\text{tea}}_T$ are the student's and the teacher's probability lists softened at temperature $T$. When $T$ is large, the two softened distributions also differ by roughly $1/T$, so the whole gradient shrinks like $1/T^2$. Multiplying by $T^2$ restores its size (Hinton, Vinyals and Dean, 2015).
The ceiling is the teacher

KL is smallest, zero, when the student matches the teacher exactly. Nothing in the loss rewards being better than the teacher. Distillation buys a smaller, cheaper model and wider coverage; it does not aim for a model that surpasses its source. (A student occasionally edges past a noisy teacher by averaging out its mistakes. That is a side effect, not the objective.)

9

Watch out for

The teacher is the ceiling
The student can approach the teacher but has no signal to pass it. Teacher mistakes are copied faithfully.
Logits need open weights
The best teachers are often closed and return text, at most a few top log-probabilities per token. There is a real trade-off between the strongest teacher and access to its probabilities.
Tokenizer mismatch
Logit distillation compares distributions token by token, so it is simple only when both models share a vocabulary (DistillationTrainer needs that). Otherwise you need an alignment method, which GOLDTrainer provides.
Too big a gap
A very small student cannot fit a very large teacher's behaviour. A mid-sized teacher sometimes teaches better than the largest one.
Exposure bias, if you train offline
Trained only on the teacher's histories, the student never practises its own. On-policy distillation (the student writes, the teacher grades each token) fixes this. It is what DistillationTrainer and GOLDTrainer do, at the cost of running both models during training.
Licence terms
Some providers forbid using their outputs to train competing models. Check before you build a dataset.

Bake a long prompt into the weights

Context distillation Prompt distillation · context baking

The teacher reads a long instruction prompt; the student reads only the short question and learns to answer as if it had read everything. Afterwards you drop the long prompt and serve a much cheaper request.

The same model with and without the long prompt, and how the student learns to answer without it.

Changes vs plain SFT: The teacher sees a long context; the student sees a short prompt
Fixes: Cost and speed no weakness. Done carelessly it removes the ability to change behaviour through the prompt
Cost: ●●●●● teacher passes + SFT
Library: Ready (exp.): SDFTTrainer, the student sees prompt, the teacher also sees privileged_context

1

The idea in plain words

Many applications send the same long block of text with every request: a persona, a style guide, a handbook, a dozen worked examples. The model rereads it every time, and you pay for it every time.

Context distillation moves that block into the weights. The teacher, often the very same model, answers with the full prompt. The student gets only the short live part and is trained to produce the same answer. After training, the short prompt alone gives the long-prompt behaviour.

Only stable text can be baked in. Anything that changes per user, per day or per deployment must stay in the live prompt, or the model will keep using the version it saw during training.

2

See it work

Choose what to bake in

This is the prompt of a library help-desk assistant. Move each block into the weights or keep it live, and watch the request shrink. Then try baking a block that changes, and read the warning.

3

Where the training signal comes from

4

A small example

Teacher: reads everything

Long context · teacher onlyYou are Pip, the help desk of the Riverside Library. Answer in at most two sentences, warmly, and always end by suggesting one related book from the catalogue. [… about 16,000 more tokens: the handbook, twelve example answers …]

UserDo you have books about volcanoes?

Teacher's answerYes, our science shelf in aisle 4 has volcano books for all ages. If you enjoy those, try “Island of Fire” next.

Student: reads only the question

User · the student's whole promptDo you have books about volcanoes?

Student · beforeVolcanoes are openings in the Earth's crust where molten rock, ash and gases escape. There are about 1,350 potentially active volcanoes worldwide…

Student · after (target = teacher's answer)Yes, our science shelf in aisle 4 has volcano books for all ages. If you enjoy those, try “Island of Fire” next.

5

What a training batch looks like

Context · teacher only, dropped from the row	Student prompt	Target · the teacher's answer
Pip guide + handbook (16k)	Do you have books about volcanoes?	Yes, our science shelf in aisle 4 has volcano books for all ages. If you enjoy those, try “Island of Fire” next.
Pip guide + handbook (16k)	Can I return a book late?	You can: there's a small fee of 10p a day, and renewing online avoids it. While you're here, “The Time Thief” is a fun read about being late!
Pip guide + handbook (16k)	Is there a quiet place to study?	Yes, the reading room on the first floor is a silent zone with desks and sockets. For a study boost, try “Make It Stick”.

# The trainer runs both sides. The student writes its own answer from "prompt"; the teacher
# (by default the frozen starting model) re-reads that answer with "privileged_context" added,
# and the student is pulled toward the teacher's token probabilities: the on-policy form in the loss.
{"prompt": [{"role": "user", "content": "Do you have books about volcanoes?"}],
 "privileged_context": "You are Pip, the help desk of the Riverside Library. Answer in at most two sentences ... [16k tokens]"}
{"prompt": [{"role": "user", "content": "Can I return a book late?"}],
 "privileged_context": "You are Pip, the help desk of the Riverside Library. Answer in at most two sentences ... [16k tokens]"}

6

The loss

$$\htmlClass{term term-1}{\mathcal{L}_{\text{ctx}}(\theta)} = -\,\mathbb{E}_{\htmlClass{term term-2}{x}}\;\mathbb{E}_{y\sim \htmlClass{term term-3}{p_{\text{tea}}(\cdot\mid c,\,x)}}\Big[\htmlClass{term term-4}{\log p_\theta(y\mid x)}\Big]$$

Read it aloud: “Over many prompts $x$, let the teacher write an answer $y$ while reading the long context $c$. Then make that answer as likely as possible for the student, who reads only $x$.” The whole technique is the missing $c$ on the student's side. It is exactly the plain SFT loss of #1; only the source of the target answer changed.

Token-level version, when the teacher's probabilities are available

$$\mathcal{L}_{\text{ctx-KL}}(\theta) = \mathbb{E}_{x}\;\mathbb{E}_{y\sim p_{\text{tea}}(\cdot\mid c,\,x)}\sum_{t}\htmlClass{term term-6}{\mathrm{KL}}\Big(p_{\text{tea}}\big(\cdot\mid \htmlClass{term term-5}{c},\, x,\, y_{<t}\big)\;\Big\|\;p_\theta\big(\cdot\mid x,\, y_{<t}\big)\Big)$$

Read it aloud: “The teacher writes the answer as before. Now, at every position $t$ of that answer, compare the teacher's whole list of next-token probabilities with the student's whole list, and add up the gaps.” The student copies not only the word the teacher picked but how sure it was of every alternative.

One-step, on-policy version (what SDFTTrainer does): the student writes, the teacher grades

$$\mathcal{L}_{\text{ctx-on}}(\theta) = \mathbb{E}_{x}\;\mathbb{E}_{\htmlClass{term term-8}{y\sim p_\theta(\cdot\mid x)}}\sum_{t} \htmlClass{term term-7}{D}\Big(p_{\text{tea}}\big(\cdot\mid \htmlClass{term term-5}{c},\, x,\, y_{<t}\big)\;\Big\|\;p_\theta\big(\cdot\mid x,\, y_{<t}\big)\Big)$$

The only change is who writes $y$: the student itself, so it also learns on its own histories. $D$ is a divergence such as KL.

Read it aloud: “The student writes its own answer from $x$ alone. The teacher re-reads that same answer with $c$ added and says, at every position, what it would have predicted next. Pull the student's prediction toward the teacher's.”

one row All three formulas are for one row: one prompt $x$ with one answer $y$. $\mathbb{E}_x\,\mathbb{E}_y$ means “averaged over prompts and their answers”, which in practice is the average over the rows of a batch. $\mathcal{L}_{\text{ctx-KL}}$ and $\mathcal{L}_{\text{ctx-on}}$ add up over the positions $t$ of that row's answer. Station 7 follows one full training step, from the batch to the update.

Every symbol · the three formulas share most of them · point at a row to light it up

Symbol	What it means
The inputs
$x$	dataThe short, live prompt, for example “Do you have books about volcanoes?”. The student sees it during training and at use time; the teacher sees it too.
$c$	dataThe long, stable context: the persona, the handbook, the worked examples (about 16,000 tokens here). Only the teacher reads it. In `SDFTTrainer` it is the `privileged_context` column.
$y$	generatedAn answer, as a list of tokens. It is not in your dataset: in the first two formulas the teacher writes it, in the third the student writes it, while training runs.
$t$	mathA position in the answer: $t = 1$ is its first token.
$y_{<t}$“y before t”	mathThe tokens of $y$ written before position $t$: the history both models are reading when they predict token $t$.
The two models
$\theta$“theta”	trainedThe student's weights, the only thing training changes.
$p_\theta$“p theta”, the student	trainedThe student model: given some text, it returns a probability for every possible next token.
$p_{\text{tea}}$“p teacher”	frozenThe teacher model. Its weights never change. Usually it is the starting copy of the same model; what makes it a teacher is only that it reads $c$.
$\mid$“given”	mathEverything to the right of the bar is what the model has read. $p_{\text{tea}}(\cdot \mid c, x)$ has read $c$ and $x$; $p_\theta(\cdot \mid x)$ has read only $x$. That missing $c$ is the technique.
$\cdot$“dot”, a blank	mathA placeholder for “every possible value”. $p_\theta(\cdot \mid x)$ is not one number but the whole list of probabilities, one for every answer (or, with $y_{<t}$ after the bar, one for every possible next token).
$p_{\text{tea}}(\cdot\mid c,x)$	frozenThe teacher's answer distribution with the long context: how likely it is to write each possible answer.
$p_{\text{tea}}(\cdot\mid c, x, y_{<t})$	frozenThe teacher's probability list for the next token at position $t$, having read the context, the prompt and the answer so far. Example: “Yes” 0.80, “We” 0.15, “Volcanoes” 0.05.
$p_\theta(\cdot\mid x, y_{<t})$	trainedThe student's probability list for the same next token, having read the same prompt and the same answer so far, but not $c$.
$\log p_\theta(y\mid x)$	trainedThe student's log-probability of the whole answer $y$, read without the context: the sum of $\log p_\theta(y_t \mid x, y_{<t})$ over its tokens, exactly as in #1.
Averages, sums and gaps
$y \sim p$“y drawn from p”	mathThe tilde means “is sampled from”: $y$ is written by that model, one token at a time.
$y\sim p_\theta(\cdot\mid x)$	trainedIn the third formula the answer is written by the student, from $x$ alone. This is what “on-policy” means: training on text the model itself produces.
$\mathbb{E}_x$“expected value over x”	mathThe average over many prompts: in practice, the average over the prompts in a batch.
$\mathbb{E}_{y\sim p}$	mathThe average over the answers that model might write. In practice you sample one answer per prompt and use it; over many steps that averages out.
$\big[\;\cdot\;\big]$	mathThe brackets hold the quantity being averaged.
$\sum_t$	mathAdd up over every token position of the answer. (Trainers usually divide by the number of tokens too, as in #1.)
$\mathrm{KL}(q \,\\|\, p)$“KL divergence of q from p”	mathA gap between two probability lists: $\mathrm{KL}(q\,\\|\,p) = \sum_v q(v)\,\log\frac{q(v)}{p(v)}$, summed over every token $v$ of the vocabulary. It is 0 when the lists are identical and grows as they differ. The first list (here the teacher's) decides which tokens matter most.
$\\|$“double bar”	mathSeparates the two lists being compared: the target (teacher, left) and the one being trained (student, right).
$D(\cdot \,\\|\, \cdot)$	mathAny divergence: a number that is 0 when the two lists match and grows as they differ. KL is the usual choice; some trainers use the reverse KL or a blend of both (Jensen–Shannon).
$\mathcal{L}_{\text{ctx}},\ \mathcal{L}_{\text{ctx-KL}},\ \mathcal{L}_{\text{ctx-on}}$	resultThe three losses: one number each, lower is better. The subscript names the version. The first carries a minus sign because a log-probability is negative; the other two are already positive because a divergence is never below 0.

What you must supply: only prompts $x$ plus the context $c$. You need no answers: a model writes them during training (the teacher in versions 1 and 2, the student in version 3). That is why context distillation is cheap to set up.

How each version works, step by step

Version 1 · $\mathcal{L}_{\text{ctx}}$: generate, then plain SFT

Give the teacher $c + x$ and let it write an answer $y$. Do this once, for every prompt, before training starts.
Build ordinary SFT rows: prompt $x$ (without $c$), target $y$.
Train the student with the #1 loss: $-\log p_\theta(y \mid x)$, averaged over the answer tokens. The student is pushed to write, from $x$ alone, what the teacher wrote with $c$.

first answer token, after "Do you have books about volcanoes?" teacher (read c) picked: "Yes" student (no c) gives: "Yes" 0.20 → loss at this position −log 0.20 = 1.61 after training: "Yes" 0.75 → loss at this position −log 0.75 = 0.29

Version 2 · $\mathcal{L}_{\text{ctx-KL}}$: match the whole probability list

Same answer $y$, written by the teacher.
At each position $t$, run both models on the same history $y_{<t}$: the teacher with $c$, the student without. Each gives a full list over the vocabulary.
Measure the gap with KL and add it up over positions. A list carries more information than the one picked token (“Yes” was likely, “We” was a reasonable second, “Volcanoes” was unlikely), so the student learns more per example.

"Yes" "We" "Volcanoes" teacher q (read c) 0.80 0.15 0.05 student p (no c) 0.20 0.20 0.60 KL(q‖p) = 0.80·log(0.80/0.20) + 0.15·log(0.15/0.20) + 0.05·log(0.05/0.60) = 1.109 − 0.043 − 0.124 = 0.94 student after training: 0.75 0.20 0.05 → KL = 0.052 − 0.043 + 0 = 0.01 (nearly matched)

Version 3 · $\mathcal{L}_{\text{ctx-on}}$: the student writes, the teacher grades

The student writes its own answer from $x$ alone, for example “Volcanoes are openings in the Earth's crust…”.
The teacher reads $c + x$ and that same student answer, and at each position gives the list it would have predicted next.
The divergence $D$ pulls the student's list toward the teacher's, position by position, on the student's own text.
Repeat with fresh student answers every step. As the student improves, its answers drift toward the teacher's style and the gaps shrink to zero.

Why bother: in versions 1 and 2 the student only ever practises on the teacher's clean answers. At use time it reads its own answers, mistakes included, and has never practised there (weakness W2). Version 3 trains exactly on those histories.

	Who writes the answer	What the teacher gives	Signal per token
$\mathcal{L}_{\text{ctx}}$	teacher, once, before training	only its text	one token: the one it picked
$\mathcal{L}_{\text{ctx-KL}}$	teacher	its full probability lists	the whole list
$\mathcal{L}_{\text{ctx-on}}$	student, fresh every step	its full lists, on the student's text	the whole list, on the student's own histories

7

One training step on the GPU

Version 1 is plain SFT on answers the teacher wrote before training, so its step is #1's. Versions 2 and 3 run the teacher on every batch, with the context c added to its input; version 3 also lets the student write the answers first, at every step. Plain SFT's station 7 walks through the full step with every number; this one shows what changes.

1BatchB prompts with answers: the teacher's (v1, v2) or freshly written by the student (v3)

2Forwardstudent reads x; teacher reads c + x, frozen

3Loss−log p (v1) or a KL per position (v2, v3), averaged

4Backwardp − y (v1) or p − q (v2, v3) at each answer token; student only

5Updatethe student moves; v3 then writes new answers

One step, slope by slope

The student never reads the context c; only the teacher does. Take one question x whose answer has 10 tokens.

one row: question x + a 10-token answer → ONE training step 0 before v1, v2: the teacher, reading c + x, wrote the answer before training v3: the student writes the 10 tokens now, from x alone, one forward pass per token 1 forward student: x + answer, one pass, every position at once, with gradient teacher (v2, v3): c + x + answer, one pass, frozen → its list q at each answer position 2 loss v1: −(1/10) Σ log p(target) v2, v3: Σ KL(q_t ‖ p_t), added over the 10 3 backward 10 slopes at the same time, one per answer token, student only: v1: g_t = (1/10)(p_t − y_t) v2, v3: g_t = p_t − q_t the positions are added, not averaged: no 1/10 each one from its own position only 4 add each weight of the student sums the 10 contributions → 1 gradient per weight 5 update the student moves once; the teacher never does

c is in no slope. It changes the teacher's lists q, and through them the targets. The student's slopes are computed on x alone.
Writing goes one token at a time, training does not. In v3 the student writes its answer token by token; the training step then reads the finished answer all at once. The sampled tokens count as fixed data: no slope flows through the sampling.
Not one after the other. The 10 slopes are measured on the same old weights and added into each weight; the weights move once, as in #1.
In a batch of B rows every slope is also divided by B.

1The batch

Versions 1 and 2: each row is a prompt x with the answer the teacher wrote, reading c, before training. Version 3: at the start of every step the student writes B answers from x alone with its current weights, and those answers are the batch.

2Forward pass

The student runs #1's forward pass on x + y, with gradient. In versions 2 and 3 the teacher runs on c + x + y, frozen and with no gradient, and hands over its probability list at every answer position. The two sequences differ in length by c, but their answer tokens line up: the teacher's list at answer token t is compared with the student's at the same token.

3Loss: from each row to one number

Each row's loss: −log p averaged over the answer (version 1) or the KL added up over the answer positions (versions 2 and 3). The batch averages the rows.

version 2, four rows (illustrative) row Σ_t KL over the row's answer 1 0.94 2 0.40 3 0.10 4 0.36 batch loss = (0.94 + 0.40 + 0.10 + 0.36) / 4 = 0.45

4Backward pass: the slope at every answer token

$$\text{v1: }\; g_t = \frac{1}{B\,|y|}\,(p_t - y_t) \qquad\qquad \text{v2, v3: }\; g_t = \frac{1}{B}\,(p_t - q_t)$$

($y_t$ is 1 at the target token and 0 at every other word.) With KL as the gap, the slope at each answer token is the student's list minus the teacher's: every word the student rates above the teacher goes down, every word below goes up. The formula adds the positions instead of averaging them, hence no $1/|y|$. Below the logits, the student's backward pass is #1's; the teacher has none. In version 3 the sampled answer counts as fixed data: no gradient flows through the sampling. (With reverse KL or Jensen–Shannon, possible in version 3, the slope is a different expression, still zero when the two lists match.)

the first answer token after "Do you have books about volcanoes?" (version 2) "Yes" "We" "Volcanoes" teacher q (read c) 0.80 0.15 0.05 student p (no c) 0.20 0.20 0.60 slope p − q −0.60 +0.05 +0.55 raised lowered lowered hard

5Update

Only the student moves. In version 3 the next step begins with the student writing new answers with the new weights.

On a real GPU

Version 1 costs what #1 costs, plus one generation run over the prompts before training.
Versions 2 and 3 add the teacher's forward pass to every step. It reads c too, often thousands of tokens, so it can cost more than the student's pass even when it is the same model.
Version 3 is the slowest. Every step starts with generation, one forward pass per token of every answer, before the teacher's pass and the student's forward and backward passes can run. Its teacher lists cannot be stored in advance, because the answers change every step.

8

The maths behind it

What the student is really learning

Training pushes the student toward $p_\theta(y\mid x) \approx p_{\text{tea}}(y\mid c, x)$ ($\approx$ reads “roughly equal to”): the student without the context answers like the teacher with it. Because $c$ is the same in every row, its effect on the answer is a fixed function of the question, and fixed functions are exactly what weights can absorb. The catch: the student learns this only for questions like the ones it trained on.
Why the saving is so large

Part of a model's work per request grows with the prompt length $L$, and the attention part grows with $L^2$, because every token looks at every earlier token. Shrinking the prompt cuts both.

full prompt 17,100 tokens live prompt 900 tokens (date, reader history, question) tokens per request 17,100 / 900 = 19× fewer attention work 19² ≈ 361× less memory for the prompt's cache 19× less

These two ratios bound the real saving: about 19× for the parts of the model whose work grows with length, up to 361× for attention while it reads the prompt. The total lands in between, and at this length the length-proportional parts usually dominate, so expect a saving nearer the low end.

Prompt caching also avoids recomputing a fixed prefix, but the prefix still takes memory and context window on every request. Distillation removes it.
What gets frozen

After training, $c$ is no longer an input. If the real context changes to $c'$, the student still behaves like $p(y \mid c, x)$, the old version, because nothing tells it otherwise.

baked in during training: handbook says "Late fee: 10p a day" the library changes: "Late fees abolished" student, asked about it: "There's a small fee of 10p a day…" ← the old rule, from the weights

9

Watch out for

Baked rules cannot change at run time
Keep live anything that varies per user, per day or per deployment: dates, account data, the output schema, the tool list, permissions. Bake only the text that is the same for every request.
Only the training distribution
The student copies the long prompt's effect on questions like the training ones. Unusual questions get weaker long-prompt behaviour. Cover the space with varied prompts.
Less steerable
Operators used to edit the prompt to change behaviour. Baked behaviour needs a retrain to change.
Failures get baked too
If the long prompt sometimes fails, those failures become permanent. Filter the teacher's answers first.
How to test it
On held-out questions, compare the student with the short prompt against the teacher with the long prompt. They should agree.

One model, many jobs

Multi-task SFT Mixed-task instruction tuning

Train one model on every kind of request it will serve, mixed into one dataset: short labels, translations, summaries, step-by-step maths. The main craft is the mixing ratio.

Two tasks, one weighted loss, and what the mixing ratio does to the small task.

Changes vs plain SFT: Which tasks are mixed, and in what proportion
Fixes: Drift on the tasks you did not train no W, and gains from shared skills
Cost: ●●●●● data work per task
Library: Build it: one mixed dataset → SFTTrainer

1

The idea in plain words

Fine-tune a model only on maths and it gets better at maths, while it can quietly get worse at everything else: answers drift in format, tone or quality. A model usually serves many kinds of request, so train it on all of them at once.

Each row keeps its own format. Related tasks share skills and help each other (positive transfer). Unrelated tasks compete for the same weights (interference). The mix decides who wins.

There is a trap. When the loss is averaged per token, tasks with long answers produce most of the learning signal and short-answer tasks get a small share of the updates, so they can end up under-trained. The lab shows by how much.

2

See it work

The mixing desk

Four tasks with very different answer lengths. Set how many rows each has, how tasks are sampled, and how the loss is averaged. Compare the share of rows with the share of learning signal.

3

Where the training signal comes from

4

A small example: one row per task

These rows are kept short to fit the page. In real data the gap is much wider: the lab uses typical averages of 2, 25, 120 and 300 answer tokens for the four tasks.

Label

UserSentiment, positive or negative: “The battery died after an hour.”

Targetnegative

Translation

UserTranslate to Spanish: “Where is the train station?”

Target¿Dónde está la estación de tren?

Summary

UserSummarise in one sentence: “Our café will close at 3 pm on Friday for staff training. Normal hours resume on Saturday at 8 am.”

TargetThe café closes early, at 3 pm, on Friday for training and reopens on Saturday at 8 am.

Step-by-step maths

UserA train travels 120 km in 1.5 hours. What is its average speed?

TargetSpeed = distance ÷ time = 120 ÷ 1.5 = 80. The average speed is 80 km/h.

5

What a training batch looks like

Task	Prompt	Target	Answer tokens
maths	A train travels 120 km in 1.5 hours. Average speed?	Speed = distance ÷ time = 120 ÷ 1.5 = 80. The average speed is 80 km/h.	31
label	Sentiment: “The battery died after an hour.”	negative	2
translate	Translate to Spanish: “Where is the train station?”	¿Dónde está la estación de tren?	12
label	Sentiment: “Best pizza I've had in years.”	positive	2
summary	Summarise in one sentence: “Our café will close at 3 pm on Friday…”	The café closes early, at 3 pm, on Friday for training and reopens on Saturday at 8 am.	24
translate	Translate to Spanish: “I would like a coffee, please.”	Quisiera un café, por favor.	9

# mixed.jsonl: the task tag is optional; it helps you sample and evaluate per task
{"task": "maths", "messages": [{"role": "user", "content": "A train travels 120 km in 1.5 hours. What is its average speed?"}, {"role": "assistant", "content": "Speed = distance ÷ time = 120 ÷ 1.5 = 80. The average speed is 80 km/h."}]}
{"task": "label", "messages": [{"role": "user", "content": "Sentiment, positive or negative: \"The battery died after an hour.\""}, {"role": "assistant", "content": "negative"}]}
{"task": "translate", "messages": [{"role": "user", "content": "Translate to Spanish: \"Where is the train station?\""}, {"role": "assistant", "content": "¿Dónde está la estación de tren?"}]}

6

The loss

$$\htmlClass{term term-1}{\mathcal{L}_{\text{multi}}(\theta)} = \htmlClass{term term-2}{\sum_{k=1}^{K}}\;\htmlClass{term term-3}{w_k}\;\htmlClass{term term-4}{\mathcal{L}_{\text{SFT}}(\theta;\,\mathcal{D}_k)}, \qquad \htmlClass{term term-3}{w_k} = \frac{\htmlClass{term term-5}{n_k}^{\,\htmlClass{term term-6}{\alpha}}}{\sum_{j} n_j^{\,\alpha}}$$

Read it aloud: “Compute the ordinary SFT loss on each task, weight each one, and add them up. The weight of a task is its row count raised to the power $\alpha$, as a share of all tasks.” Point at a row of the table to light its term up in the formula.

whole tasks This formula is written over whole tasks: $\mathcal{L}_{\text{SFT}}(\theta;\mathcal{D}_k)$ is the average SFT loss over every row of task $k$. The loss of one row is the plain #1 loss; in practice the weights $w_k$ act by deciding how often each task's rows reach a batch, which on average equals multiplying the task losses by $w_k$. Station 7 follows one full training step, from the batch to the update.

Every symbol in the formula

Symbol	What it means
$\mathcal{L}_{\text{multi}}(\theta)$“L-multi of theta”	resultThe total loss over all tasks: one number, lower is better.
$\theta$“theta”	trainedThe weights of the one model that learns every task.
$K$	dataHow many tasks you mix: 4 on this page (label, translation, summary, maths).
$k$, $j$	mathCounters that name one task: $k = 1$ is the first task, $k = K$ the last. $j$ is a second counter, used to add up over all tasks inside the weight.
$\sum_{k=1}^{K}$“sum for k from 1 to K”	mathAdd up the term on its right once for each task.
$\mathcal{D}_k$“D sub k”	dataThe rows of task $k$: for example all the translation rows.
$\mathcal{L}_{\text{SFT}}(\theta;\,\mathcal{D}_k)$	trainedPlain SFT loss (#1) on the rows of task $k$, averaged per row. The semicolon reads “measured on”. Most trainers instead divide by all answer tokens in the batch, which adds the hidden length weight shown in step 1 below.
$n_k$“n sub k”	dataHow many rows task $k$ has: for example 50,000, 5,000 and 500.
$\alpha$“alpha”, the mixing exponent	settingA number you choose between 0 and 1. $\alpha = 1$: proportional to size, big tasks dominate. $\alpha = 0$: every $n_k^{0} = 1$, so every task gets the same share. In between (often 0.3 to 0.7): a compromise.
$n_k^{\,\alpha}$“n-k to the power alpha”	mathThe row count raised to the power $\alpha$. With $\alpha = 0.5$ it is the square root: $\sqrt{50{,}000} = 224$, $\sqrt{500} = 22$. Powers below 1 shrink big numbers much more than small ones, so the gap between tasks narrows.
$\sum_j n_j^{\,\alpha}$	mathThe same quantity added up over all tasks. Dividing by it turns the numbers into shares that add up to 1.
$w_k$“w sub k”, the weight	resultThe weight of task $k$: its share of the training draws, between 0 and 1. The weights add up to 1. In practice you usually sample rows with these shares rather than multiply the losses; on average the two are the same.

What you must supply: the rows of each task, ideally tagged with their task so you can count $n_k$, sample by task and evaluate each task separately. You choose $\alpha$.

How the loss works, step by step

Count the rows of every task: $n_1, n_2, \dots, n_K$.
Turn the counts into weights with the exponent $\alpha$: $w_k = n_k^{\alpha} / \sum_j n_j^{\alpha}$.
Fill each batch by drawing rows from task $k$ with probability $w_k$, shuffled together.
Compute the ordinary #1 loss on every row. Averaged over many batches, task $k$ contributes $w_k$ times its own SFT loss.
Step downhill. The one set of weights $\theta$ moves to serve all tasks at once, each in proportion to its weight.

Worked example: three tasks with 50,000, 5,000 and 500 rows, and per-task losses 0.40, 0.90 and 1.60 (illustrative)

big task medium task small task rows n_k 50,000 5,000 500 SFT loss on the task 0.40 0.90 1.60 α = 1 weights 0.901 0.090 0.009 w_k × loss 0.360 0.081 0.014 L_multi = 0.46 α = 0.5 weights 0.706 0.223 0.071 w_k × loss 0.282 0.201 0.113 L_multi = 0.60 α = 0 weights 0.333 0.333 0.333 w_k × loss 0.133 0.300 0.533 L_multi = 0.97

The small task has the worst loss, 1.60, but with $\alpha = 1$ it adds only 0.014 to the total: training barely notices it and leaves it unlearned. Lowering $\alpha$ gives it a louder voice. The totals are not comparable across $\alpha$; what changes is which task the gradient listens to.

7

One training step on the GPU

The training step is #1's. The task weights act before it, in the sampler that fills each batch. Plain SFT's station 7 walks through the full step with every number; this one shows what changes.

1BatchB rows drawn from task k with probability w_k

2Forwardas #1, all tasks in one grid

3Loss#1's loss, a plain average over the mixed rows

4Backwardp − y at each answer token; the tasks meet in the weights

5Updateone step serves every task

One step, slope by slope

#1's step on a batch that mixes tasks. Take the batch of 10 rows of point 3 below: 7 from the big task, 2 from the medium one, 1 from the small one, each with 10 answer tokens.

10 rows from three tasks, 10 answer tokens each → ONE training step 0 before the sampler filled each place from task k with probability w_k 1 forward one pass over the 10 rows, every task in the same grid 2 loss #1's loss on each row, a plain average over the 10 rows 3 backward 100 slopes at the same time, one per answer token: g_t = k (p_t − y_t) k = 1/(10 × 10) = 1/100, whatever the token's task each one from its own p and its own target only 4 add each weight sums all 100 contributions, from every task → 1 gradient per weight 5 update every weight moves once, for every task at once

w_k decides how many slopes a task brings, not how hard each one pushes. The big task brings 70 slopes, the small one 10, and every one has k = 1/100.
Where tasks disagree, they cancel in the sum. If two tasks push a weight in opposite directions, their contributions are added first, and only the total moves it.
In “See it in action” the toy multiplies each task's loss by w_k instead, with one row per task: k = 0.75 for task A and 0.25 for task B. On average the two give the same step; inside one step, sampling gives every token the same k.
Not one after the other. The 100 slopes are measured on the same old weights and added into each weight; the weights move once, as in #1.

1The batch

Each of the B places is filled from task k with probability $w_k$, so a batch mixes tasks. Nothing in the batch says which task a row came from, apart from its text.

2Forward pass

#1's, unchanged.

3Loss: from each row to one number

Each row gets #1's loss and the batch takes a plain average. Task k fills about a share $w_k$ of the places, so over many batches this average equals $\sum_k w_k\,\mathcal{L}_{\text{SFT}}(\theta;\mathcal{D}_k)$.

α = 0.5, a batch of 10 rows, the task losses of the example above weights 0.706 / 0.223 / 0.071 → about 7 big-task rows, 2 medium, 1 small batch loss = (7 × 0.40 + 2 × 0.90 + 1 × 1.60) / 10 = 0.62 the formula's 0.60, up to rounding to whole rows

4Backward pass: the slope at every answer token

($y_t$ is 1 at the target token and 0 at every other word.) Each answer token gets #1's slope, $\frac{1}{B\sum m_i}(p_t - y_t)$. The tasks meet one step further down: every weight adds up the pushes of every row, from every task. When two tasks want a weight to move in opposite directions, their pushes are added and the larger total wins: $w_k$ sets how many rows each task brings, and how wrong the model is on those rows sets how hard each one pushes.

5Update

One update for all tasks at once.

On a real GPU

The step costs what #1's costs. The mix lives in the data loader.
Packing. Rows from different tasks differ a lot in length, which wastes room as padding. Trainers often pack several short rows into one sequence, ideally with attention kept inside each row (some packers let it cross row boundaries unless that is switched on), so fewer positions are padding.

8

The maths behind it

Token averaging hides an extra weight

#1 divides each row by its own answer length. Trainers such as TRL's SFTTrainer instead divide by all the answer tokens in the batch, so each token, not each row, counts once. A task with long answers brings more tokens, so its share of the gradient is roughly its weight times its average answer length $\bar{L}_k$:

$$\text{share}_k = \frac{w_k\,\bar{L}_k}{\sum_j w_j\,\bar{L}_j}$$

$w_k$	resultThe weight of task $k$ from the loss above: its share of the rows drawn.
$\bar{L}_k$“L-bar sub k”	dataThe average answer length of task $k$, in tokens. The bar on top of a letter means “average”.
$w_k\,\bar{L}_k$	mathHow many answer tokens task $k$ brings to a batch, relative to the others: rows drawn times tokens per row.
$\sum_j w_j\,\bar{L}_j$	mathThe same, added up over every task: all the answer tokens in the batch.
$\text{share}_k$	resultThe fraction of the gradient that comes from task $k$ when every token counts once.

task A: 4,000-token answers task B: 40-token answers equal numbers of rows share of A = 4,000 / (4,000 + 40) = 99% share of B = 40 / (4,000 + 40) = 1% ← B gets 1% of the updates

Fixes: average per example instead of per token (a custom loss in most trainers), cap answer lengths, or multiply each task's weight by $1/\bar{L}_k$.

Temperature sampling, in numbers

row counts: 50,000 5,000 500 α = 1 (proportional) 90.1% 9.0% 0.9% α = 0.5 (square root) 70.6% 22.3% 7.1% √: 223.6, 70.7, 22.4 → sum 316.7 α = 0 (equal) 33.3% 33.3% 33.3% with α = 0 and 30,000 draws, the 500-row task gets 10,000 draws: every row is seen 20 times
Transfer and interference, as arrows

Each task pulls the weights in some direction: its gradient $g_k$. If two directions roughly agree (positive dot product), a step for one also helps the other. If they disagree (negative dot product), they fight. The dot product $g_A \cdot g_B$ multiplies the matching entries of the two arrows and adds the results: positive when they point roughly the same way, negative when they point apart, zero when they are at right angles.

g_A = (1.0, 0.5) g_B = (0.8, 0.6) g_A · g_B = 0.80 + 0.30 = +1.10 → transfer g_A = (1.0, 0.5) g_C = (−0.9, 0.2) g_A · g_C = −0.90 + 0.10 = −0.80 → interference

9

Watch out for

Format bleed
JSON leaking into prose answers, or bullet points into one-word labels. Keep each task's format consistent and clearly cued by its prompt.
Long answers drown short ones
Per-token averaging gives long-answer tasks most of the signal. Check the signal share, not only the row share.
One bad run breaks everything
A single model for all tasks means a bad fine-tune regresses all of them together. Evaluate every task separately before shipping.
Imbalance inside a task
A yes/no task that is 95% “no” can learn to always answer “no” and still score 95%. Check per-class accuracy.
Small tasks get memorised
Up-weighting a tiny task repeats its rows many times. Add more data or stop at a lower weight.

Teach the way back after a mistake

Correction and recovery SFT Offline, DAgger-style data

Plain SFT only shows perfect paths, so after one slip the model is lost. This technique adds examples that start right after a mistake and show the right next step, so the model learns to get back on track.

A slip stays in the history, read but never graded, and the row shows the way back.

Changes vs plain SFT: Who writes the answers on bad steps, and which situations appear in the data
Fixes: W2 small mistakes snowball. The only SFT variant aimed at it
Cost: ●●●●● a judge per step + validation
Library: Build it: corrected targets → SFTTrainer

1

The idea in plain words

A solution, a conversation or an agent's run is a chain of steps. Plain SFT trains on demonstrations: the histories are the demonstrator's own, and after filtering (#2) every step in them is right, so every situation in the data is “on track”. When the model makes its first mistake at use time, it lands in a situation it has never seen. With no example of what to do there, it tends to make another mistake, and errors pile up.

Correction SFT adds two kinds of rows:

Corrections: for a step that went wrong, the right step to take instead, from the same situation.
Recoveries: rows whose history already contains a mistake, and whose target notices it and repairs it.

The key detail: the mistake sits in the prompt part of the row, which is never graded. The model learns what to do after a mistake without ever being trained to make one.

2

See it work

One slip, two models

The shaded lane marks situations that appear in the training data. Choose where the slip happens and compare a model trained with and without recovery rows. Below, see how the damage grows with the length of the task.

3

Where the training signal comes from

Three jobs: find the mistake, write the fix, check the fix.

4

A small example

The recorded run, step by step

UserA recipe needs 250 g of flour for 10 cookies. How much flour for 24 cookies?

Step 1 · judged goodFlour per cookie: 250 ÷ 10 = 25 g.

Step 2 · judged bad (first error)For 24 cookies: 25 × 24 = 500 g.

Correction row: the right step, same situation

Prompt · not gradedA recipe needs 250 g of flour for 10 cookies. How much flour for 24 cookies?
Step 1: Flour per cookie: 250 ÷ 10 = 25 g.

Target · the fixFor 24 cookies: 25 × 24 = 600 g.

Recovery row: notice the mistake, repair it

Prompt · not graded, mistake includedA recipe needs 250 g of flour for 10 cookies. How much flour for 24 cookies?
Step 1: Flour per cookie: 250 ÷ 10 = 25 g.
Step 2: For 24 cookies: 25 × 24 = 500 g.

Target · the recoveryLet me check that: 25 × 20 = 500 and 25 × 4 = 100, so 25 × 24 = 600 g, not 500 g. The recipe needs 600 g of flour.

5

What a training batch looks like

Kind	History · not graded	Target · graded	Weight
good	Flour question	Flour per cookie: 250 ÷ 10 = 25 g.	1.0
correction	Flour question + step 1	For 24 cookies: 25 × 24 = 600 g.	1.0
recovery	Flour question + step 1 + wrong step 2	Let me check that: 25 × 20 = 500 and 25 × 4 = 100, so 25 × 24 = 600 g, not 500 g. …	2.0
good	“Convert 3.5 hours to minutes.”	3.5 × 60 = 210 minutes.	1.0
recovery	“Is 2025 a leap year?” + “2025 ÷ 4 = 506.25, so yes.”	Wait: 506.25 is not a whole number, so 2025 is not divisible by 4. 2025 is not a leap year.	2.0

# prompt / completion rows: everything in "prompt", the wrong assistant turn included, is masked;
# only "completion" is graded. A plain "messages" row would grade the mistake too.
{"kind": "recovery",
 "prompt": [{"role": "user", "content": "A recipe needs 250 g of flour for 10 cookies. How much flour for 24 cookies?"},
            {"role": "assistant", "content": "Flour per cookie: 250 ÷ 10 = 25 g. For 24 cookies: 25 × 24 = 500 g."}],
 "completion": [{"role": "assistant", "content": "Let me check that: 25 × 20 = 500 and 25 × 4 = 100, so 25 × 24 = 600 g, not 500 g. The recipe needs 600 g of flour."}]}

# λ = 2: SFTTrainer has no per-row weight, so write each recovery row twice (or override the loss).
# Some trainers and fine-tuning services offer per-turn switches instead; check yours.

6

The loss

$$\htmlClass{term term-1}{\mathcal{L}_{\text{corr}}(\theta)} = -\frac{1}{\htmlClass{term term-2}{Z}}\Bigg[\sum_{(s,a)\in\htmlClass{term term-3}{\mathcal{D}_{\text{good}}}}\!\!\log\pi_\theta(a\mid s) \;+\!\! \sum_{(s,a^\star)\in\htmlClass{term term-4}{\mathcal{D}_{\text{fix}}}}\!\!\log\pi_\theta(\htmlClass{term term-5}{a^\star}\mid s) \;+\; \htmlClass{term term-6}{\lambda}\!\!\sum_{(s,a)\in\htmlClass{term term-7}{\mathcal{D}_{\text{rec}}}}\!\!\log\htmlClass{term term-8}{\pi_\theta(a\mid s)}\Bigg]$$

Read it aloud: “Three piles of rows, all with the ordinary SFT loss: the good steps, the bad steps with their target replaced by the fix, and the recoveries counted $\lambda$ times. Divide by the total weight.” Point at a row of the table to light its term up in the formula.

whole set This formula is written over all the rows at once. The loss of one row is its #1 loss times its weight: 1 for good and fix rows, $\lambda$ for recovery rows. $Z$, the sum of those weights, makes the whole thing a weighted average. Station 7 follows one full training step, from the batch to the update.

Every symbol in the formula

Symbol	What it means
Agent notation: new letters for prompt and answer
$s$“s”, the situation (state)	dataEverything the model has seen so far: the question plus the earlier steps of the run. It plays the role of the prompt $x$. In a recovery row it includes the mistake.
$a$“a”, the action	dataThe next step to write, for example “Flour per cookie: 250 ÷ 10 = 25 g.” It plays the role of the answer $y$.
$a^\star$“a star”	dataThe corrected step, written and checked for a situation where the recorded step was wrong: “25 × 24 = 600 g”. The star marks “the right one”.
$\pi_\theta(a\mid s)$“pi theta of a given s”	trainedThe probability the model gives to step $a$ in situation $s$. $\pi$ (pi) is the usual name for a model that picks actions, a policy; it is the same thing as $p_\theta(y \mid x)$.
$\theta$	trainedThe model's weights, the only thing training changes.
The three piles of rows
$\mathcal{D}_{\text{good}}$“D good”	dataSteps the judge marked good, kept as they are, as in #2.
$\mathcal{D}_{\text{fix}}$“D fix”	dataCorrection rows: the situation $s$ before a bad step is kept; the bad step is thrown away and $a^\star$ becomes the target.
$\mathcal{D}_{\text{rec}}$“D rec”, recoveries	dataRecovery rows: the history $s$ already contains a mistake, and the target $a$ notices and repairs it.
$\sum_{(s,a)\in\mathcal{D}}$	mathAdd the term up over every row of that pile.
Weights and the result
$\lambda$“lambda”	settingHow much more a recovery row counts than an ordinary row, for example 2. Recoveries are rare and valuable, so each one is counted $\lambda$ times.
$Z$“Z”, the normaliser	mathThe total weight, $\|\mathcal{D}_{\text{good}}\| + \|\mathcal{D}_{\text{fix}}\| + \lambda\|\mathcal{D}_{\text{rec}}\|$, counted in rows ($\|\cdot\|$ is “how many rows”). Dividing by it makes the loss an average per row. Each row's $\log\pi_\theta$ sums its tokens; trainers usually average them per token instead, as in #1.
$\log$, $-$	mathAs in #1: the log turns a probability into a negative score, the minus sign makes the loss positive.
$\mathcal{L}_{\text{corr}}(\theta)$“L-corr of theta”	resultThe loss over all three piles: one number, lower is better.

What you must supply: recorded runs split into steps, a judgement for every step (good, or the first error), a correct step $a^\star$ for each error, and repair texts for the recovery rows. You choose $\lambda$. The mistakes themselves stay inside $s$, on the masked, never-graded side.

How the loss works, step by step

Sort the steps of each run. Good steps go to $\mathcal{D}_{\text{good}}$. At the first bad step, keep the situation and write the right step: a row for $\mathcal{D}_{\text{fix}}$. Keep the bad step in the history and write a repair: a row for $\mathcal{D}_{\text{rec}}$.
Compute the #1 loss on every row. The situation $s$ is the masked prompt, so only the target step is graded.
Weigh and add. Good and correction rows count once, recovery rows count $\lambda$ times. Divide by $Z$.
Step downhill. The model learns to take the right step from clean situations and to repair from situations that already contain a slip, which plain SFT never shows it.

Worked example: the five rows of the batch above, $\lambda = 2$ (per-row losses are illustrative)

row pile weight −log π (per token) weight × loss flour, step 1 good 1 0.20 0.20 flour, step 2 fixed fix 1 1.30 1.30 flour, repair after 500 g rec 2 2.10 4.20 hours to minutes good 1 0.10 0.10 leap year, repair rec 2 1.80 3.60 Z = 2 good + 1 fix + 2 × 2 rec = 7 L_corr = 9.40 / 7 = 1.34

The recovery rows have the highest losses: before training, the model has no idea how to get back on track, which is exactly the lesson. With $\lambda = 2$ they supply 7.80 of the 9.40, so most of this step's learning goes to recovering.

7

One training step on the GPU

The training step is #1's with one number added per row: its weight $c_i$, 1 or $\lambda$. Plain SFT's station 7 walks through the full step with every number; this one shows what changes.

1Batchgood, fix and recovery rows mixed, each with its weight

2Forwardas #1

3Lossa weighted average of the row losses

4Backward(c_i / Σc) × (p − y) / Σm at each answer token

5Updateonce; recovery rows pull λ times harder

One step, slope by slope

#1's step with one weight c_i per row. Take a batch of three rows with 10 answer tokens each: a good row, a fix row and a recovery row, with λ = 2.

good + fix + recovery row, 10 answer tokens each, λ = 2 → ONE training step 1 forward one pass over the 3 rows; the recovery row's history holds the slip 2 loss Σ c_i × (row loss) / Σ c_i weights c = 1, 1, 2 Σ c = 4 3 backward 30 slopes at the same time, one per answer token: g_t = k_i (p_t − y_t) k_i = (c_i / Σ c) × (1/10) good and fix rows: 1/4 × 1/10 = 1/40 recovery row: 2/4 × 1/10 = 1/20 each one from its own p and its own target only 4 add each weight sums the 30 contributions → 1 gradient per weight 5 update every weight moves once; the recovery tokens pushed it twice as hard

The slip makes no slope. The wrong step sits in the recovery row's history: read, never graded. Only the repair after it is graded.
c_i scales a whole row. Every token of a recovery row gets the same double share, and it does not depend on the other rows' probabilities.
Not one after the other. The 30 slopes are measured on the same old weights and added into each weight; the weights move once, as in #1.

1The batch

Rows from the three piles are shuffled together. Next to the token grid the trainer passes a list of B weights: 1 for good and fix rows, $\lambda$ for recovery rows.

2Forward pass

#1's. Recovery rows carry the slip in their history: more prompt tokens, read and never graded.

3Loss: from each row to one number

Each row gets #1's loss times its weight, and the batch divides by the sum of the weights: $\sum_i c_i\,\mathcal{L}^{(i)} / \sum_i c_i$.

the five rows of the example above as one batch, λ = 2 Σ c_i × row loss = 0.20 + 1.30 + 4.20 + 0.10 + 3.60 = 9.40 Σ c_i = 7 batch loss = 9.40 / 7 = 1.34

4Backward pass: the slope at every answer token

$$g_t = \frac{c_i}{\sum_j c_j}\cdot\frac{1}{\sum m_i}\,\big(p_t - y_t\big)$$

($y_t$ is 1 at the target token and 0 at every other word.) The same p − y as #1, scaled by the row's share of the weights. In the batch above a recovery row as a whole gets 2/7 of the push and a good row 1/7, each split equally over its answer tokens (÷ Σm_i). Below the logits, #1's backward pass.

5Update

Once, by the sum of all rows' pushes.

On a real GPU

The step costs what #1's costs: the weights are B numbers multiplied into the row losses.
Recovery rows are longer, so a batch of them needs more memory for activations.
About the same effect without weights: put each recovery row in the data $\lambda$ times.

8

The maths behind it

Why mistakes snowball: the $\varepsilon T^2$ rule

Say the model makes a mistake with a small probability $\varepsilon$ at each of $T$ steps, and that once off track it stays off, because it never saw how to get back. Expected steps spent off track:

$$\mathbb{E}[\text{steps off track}] = \sum_{t=1}^{T}\Big(1-(1-\varepsilon)^{t}\Big) \;\approx\; \frac{\varepsilon\,T^2}{2}\quad\text{when } \varepsilon T \text{ is small}$$

$\varepsilon$“epsilon”	trainedThe model's chance of making a mistake at any one step, for example 0.01 = one slip per 100 steps.
$T$	dataHow many steps the task has. (Here $T$ is a number of steps, not the temperature of #3.)
$t$	mathOne step of the run, from 1 to $T$.
$(1-\varepsilon)^{t}$	mathThe chance that all of the first $t$ steps went right: $1-\varepsilon$ multiplied by itself $t$ times.
$1-(1-\varepsilon)^{t}$	mathThe chance that at least one slip has happened by step $t$. Under “once off track, stays off” this is the chance that step $t$ is off track.
$\sum_{t=1}^{T}$	mathAdding those chances over all steps gives the expected number of off-track steps.
$\mathbb{E}[\,\cdot\,]$“expected value”	mathThe average over many runs.
$\approx \frac{\varepsilon T^2}{2}$	resultThe short-cut answer: damage grows with the square of the task length. Double the steps, four times the damage.

“Small” means well under one expected mistake per task. When $\varepsilon T$ is large, nearly every run slips early and the damage approaches $T - 1/\varepsilon$: almost the whole task is lost.

ε = 0.01 (1 mistake per 100 steps), T = 50 steps exact: 50 − 99 × (1 − 0.99⁵⁰) = 50 − 39.1 = 10.9 steps off track rough rule: 0.01 × 50² / 2 = 12.5

With recovery: the $\varepsilon T$ rule

If the model has learned to get back on track, each mistake costs about one step. The expected damage is just the expected number of mistakes:

$$\mathbb{E}[\text{steps off track}] \approx \varepsilon\,T$$

$\varepsilon$, $T$	mathAs above: the chance of a slip per step, and the number of steps.
$\varepsilon\,T$	resultThe expected number of slips in a run. With recovery each slip costs about one step, so the damage now grows only in step with the length.

ε = 0.01, T = 50 → 0.01 × 50 = 0.5 steps ← about 22× less damage than 10.9, at the same error rate

While $\varepsilon T$ is small the ratio between the two is about $T/2$, so longer tasks gain more from recovery data. Once nearly every run slips, the ratio levels off near $1/\varepsilon$.

The distribution-shift picture, and DAgger

Plain SFT learns on situations from the expert's runs, $d_{\text{expert}}$. At use time the model meets situations from its own runs, $d_{\pi_\theta}$, which include its slips. Recovery rows, and corrections of bad steps that come after an earlier slip, add training targets on off-track situations. Corrections at the first error lower $\varepsilon$; recoveries turn $\varepsilon T^2$ into $\varepsilon T$. DAgger (Ross, Gordon and Bagnell, 2011) does this properly: let the learner run, have an expert label every situation it reached, add those rows to everything collected so far, retrain, repeat. If the expert can recover from a single slip cheaply, the worst-case cost grows like $\varepsilon T$ instead of $\varepsilon T^2$ (Ross and Bagnell, 2010), with $\varepsilon$ measured on the learner's own situations. The offline version on this page uses mistakes already in the recorded data and carries no such guarantee.

Why the model never learns the mistake

In a recovery row the wrong step is in the history, where the mask is $m_t = 0$. Its tokens contribute nothing to the loss, so their gradient is zero. Here $s_t$ is the $t$-th token of the row, as in #1, not the situation $s$:

$$m_t = 0 \;\Rightarrow\; \frac{\partial\,\mathcal{L}}{\partial\,\log p_\theta(s_t\mid s_{<t})} = 0$$

$m_t$	dataThe loss mask of #1 at token $t$: 0 on the history (the mistake included), 1 on the target.
$s_t$, $s_{<t}$	dataThe $t$-th token of the row and the tokens before it.
$\log p_\theta(s_t\mid s_{<t})$	trainedThe model's log-probability of that token.
$\frac{\partial\,\mathcal{L}}{\partial\,\cdots}$“partial L over partial …”	mathThe slope: how much the loss would change if that log-probability changed a little. A slope of 0 means training gets no push to change it.
$\Rightarrow$“implies”	math“If the left side is true, the right side follows.”

The model reads the mistake as context and learns only from the repair that follows.

9

Watch out for

Knowing the future
The fix is written after the judge has seen how the run ended. It may use information that was not available at that step. Check fixes by hand and keep any reasoning grounded in what was known at the time.
Someone else's mistakes
Offline, the mistakes come from the recorded runs, not from the student. Its own slips may look different. Full DAgger needs the student to run and be corrected, an online loop.
Never put the mistake in the target
If the wrong step ends up in the graded part, you train the model to make it. Check the mask on recovery rows.
Too much recovery
With a large $\lambda$ the model may start second-guessing correct steps (“Let me check that…” everywhere). Keep recoveries a modest share.
Validate every fix
An unchecked fix that is itself wrong teaches a new mistake, exactly where the model is most fragile.

Turn a failure into a lesson for another goal

Hindsight relabelling Hindsight experience replay, for language

A failed attempt at one goal is often a perfect attempt at another. Rewrite the prompt so it asks for what the attempt actually achieved, and the failure becomes a valid training example.

A failed attempt, a rewritten goal, and the plain SFT step that follows.

Changes vs plain SFT: How a row is written: the goal in the prompt is rewritten
Fixes: Wasted data when most attempts fail no W
Cost: ●●●●● verify + rewrite each failure
Library: Build it: relabelled rows → SFTTrainer

1

The idea in plain words

When most attempts fail, most of your data gets thrown away. Hindsight relabelling recycles it: look at what the attempt actually did, verify it, and rewrite the request so it asks for exactly that. The answer stays the same; only the goal changes.

The trick comes from reinforcement learning (Hindsight Experience Replay, Andrychowicz et al., 2017), where a robot arm that missed its target still learned how to reach the spot it did reach. Its supervised form is goal-conditioned supervised learning (Ghosh et al., 2019); for language models, Hindsight Instruction Relabeling (Zhang et al., 2023) rewrites instructions to match what the model actually produced.

It turns the model into a goal-following one: it learns to act given a goal, written $\pi(a \mid s, g)$, so one attempt can teach a different goal than the one it was made for. It only helps when the achieved outcome is something a person might really ask for.

2

See it work

Relabel a failed walk

A walking assistant was asked to reach a place on a small town map and failed. Pick a run and switch between the original goal and the hindsight goal. One of the three runs should not be relabelled; find out which.

3

Where the training signal comes from

4

A small example

Original: a failure

User · original goalWrite a haiku (three lines, 5-7-5 syllables) about the sea.

Answer · fails the goal (4 lines)The waves roll in slow
silver light on every crest
gulls call from the rocks
the tide forgets the shore

Relabelled: a valid example

User · hindsight goalWrite a four-line poem about the sea.

Same answer · now correctThe waves roll in slow
silver light on every crest
gulls call from the rocks
the tide forgets the shore

The part that must be rewritten too

Reasoning kept as is · wrongI need 5-7-5 syllables, so three short lines. The waves roll in slow…

Reasoning rewritten for the new goalFour lines, calm images: waves, light, gulls, tide. The waves roll in slow…

Left unchanged, the old reasoning teaches the model to say one thing and do another.

5

What a training batch looks like

Original goal	Achieved (verified)	New prompt	Answer · unchanged	Checked by
Haiku about the sea	4 lines, 23 syllables	Write a four-line poem about the sea.	The waves roll in slow / silver light on every crest / …	line counter
Summarise in 100 words	45 words	Summarise this article in about 45 words.	City council approved the new bike lanes…	word counter
Walk to the library	ended at the bakery	Walk to the bakery.	→ → ↑ → ↑	map end state
List 10 fruits	7 fruits, all valid	List 7 fruits.	apple, pear, plum, kiwi, mango, lime, fig	count + judge

# keep the provenance beside each relabelled row; the trainer only reads "messages"
{"original_goal": "Write a haiku (three lines, 5-7-5 syllables) about the sea.", "achieved": "4 lines", "verified_by": "line_counter",
 "messages": [{"role": "user", "content": "Write a four-line poem about the sea."},
              {"role": "assistant", "content": "The waves roll in slow\nsilver light on every crest\ngulls call from the rocks\nthe tide forgets the shore"}]}

6

The loss

$$\htmlClass{term term-1}{\mathcal{L}_{\text{HER}}(\theta)} = -\,\mathbb{E}_{\htmlClass{term term-2}{\tau}\sim\mathcal{D}}\Bigg[\sum_{t}\log \htmlClass{term term-3}{\pi_\theta}\big(a_t \mid s_t,\; \htmlClass{term term-4}{g = \phi(\tau)}\big)\Bigg]$$

Read it aloud: “For every recorded attempt, find the goal it actually reached, write that goal into the prompt, and train on the attempt's steps as usual.” The only new symbol is the relabel function $\phi$. Point at a row of the table to light its term up in the formula.

one row The part inside $\mathbb{E}$ is the loss of one row: one recorded attempt $\tau$, relabelled, with its steps added up over $t$. $\mathbb{E}_{\tau\sim\mathcal{D}}$ means “averaged over attempts”, in practice over the rows of a batch. Station 7 follows one full training step, from the batch to the update.

Every symbol in the formula

Symbol	What it means
$\mathcal{L}_{\text{HER}}(\theta)$“L-HER”, hindsight experience replay	resultPlain SFT loss on relabelled attempts: one number, lower is better.
$\theta$	trainedThe model's weights, the only thing training changes.
$\tau$“tau”, a trajectory	dataOne recorded attempt: the steps $a_1, a_2, \dots$ taken in situations $s_1, s_2, \dots$ For a single answer, like the poem, it is just that answer.
$\mathcal{D}$	dataThe attempts you train on: the successes, plus the failures whose achieved goal was verified and judged realistic. Failures that reached nothing useful are left out.
$\mathbb{E}_{\tau\sim\mathcal{D}}$“average over attempts tau from D”	mathThe average over the attempts in $\mathcal{D}$; in practice, over the rows of a batch. $\sim$ reads “drawn from”.
$t$, $\sum_t$	mathA step of the attempt, and “add up over every step”.
$s_t$“s sub t”, the situation	dataWhat the model had seen before step $t$: the earlier steps and observations.
$a_t$“a sub t”, the action	dataThe step actually taken at $t$: a move, a tool call, a line of text. It is the target and is never changed; only the goal is.
$g$“g”, the goal	dataThe goal written into the prompt, for example “Write a four-line poem about the sea.”
$\phi(\tau)$“phi of tau”, the relabel function	frozenReads the attempt and returns the goal it verifiably reached (4 lines, 45 words, ended at the bakery), instead of the goal that was asked for. In practice a checker (a line counter, a word counter, the map's end state) plus a template that writes the new prompt.
$\pi_\theta(a_t \mid s_t, g)$“pi theta of a-t given s-t and g”	trainedA goal-following model (a policy): the probability of step $a_t$ in situation $s_t$ when asked for goal $g$.
$\log$, $-$	mathAs in #1: the log turns a probability into a negative score, and the minus sign makes the loss positive.

What you must supply: recorded attempts, successes and failures, plus a checker that can tell what an attempt really achieved. The answers stay exactly as recorded; only the goal in the prompt is rewritten.

How the loss works, step by step

Collect attempts, most of them failures at the goal that was asked for.
Find what each one achieved with the checker: $\phi(\tau)$. Drop attempts that achieved nothing anyone would ask for.
Rewrite the goal in the prompt to $g = \phi(\tau)$. Rewrite any reasoning that mentions the old goal, too.
Train with the #1 loss on the unchanged steps, now read under the new goal.

Why the relabel matters: the same four-line answer, trained under two goals

goal in the prompt what the gradient teaches verdict g = asked: "Write a haiku…" "asked for a haiku → write four lines" harmful g = φ(τ): "Write a four-line poem…" "asked for a four-line poem → write four lines" correct the four rows of the batch above, per-token losses 1.10, 0.70, 0.40, 0.90 (illustrative) L_HER = (1.10 + 0.70 + 0.40 + 0.90) / 4 = 0.78

The loss is computed exactly as in #1. The whole technique lives in the goal $g$ on the right of the bar: change what the model was “asked”, and the same answer goes from a bad example to a good one.

7

One training step on the GPU

The training step is #1's. The relabelling happens before training: the trainer only ever sees ordinary SFT rows. Plain SFT's station 7 walks through the full step with every number; this one shows what changes.

1BatchB relabelled rows: the goal in the prompt is the one achieved

2Forwardas #1

3Loss#1's loss, averaged

4Backwardp − y at each answer token, as #1

5Updateonce

One step, slope by slope

#1's step on the relabelled rows. Take one row: the goal the attempt actually reached, the prompt, and the attempt's 10 answer tokens.

one relabelled row: new goal + prompt + the attempt's 10 tokens → ONE training step 0 before the checker replaced the goal with the one the attempt reached, φ(τ) 1 forward one pass over the row, every position at once 2 loss −(1/10) Σ log p(target) 3 backward 10 slopes at the same time, one per answer token: g_t = k (p_t − y_t) k = 1/10 each one from its own p and its own target only 4 add each weight sums the 10 contributions → 1 gradient per weight 5 update every weight moves once

The goal makes no slope. It is a prompt token. Its embedding still gets gradient through attention, because all 10 answer tokens read it: that is how the goal becomes tied to the steps that reach it.
The original row is not in the batch. Relabelling happened before training; the failed goal sends nothing.
Not one after the other. The 10 slopes are measured on the same old weights and added into each weight; the weights move once, as in #1.

1The batch

The checker $\phi$ runs over every recorded attempt once, before training, and rewrites the goal. The batches are filled with these rows like any SFT data.

2Forward pass

#1's, unchanged.

3Loss: from each row to one number

Each row gets #1's loss; the batch averages them. (The formula adds up the steps of an attempt; trainers usually average per answer token, as in #1, which changes the scale and also how much long attempts weigh against short ones.)

the four rows of the example above as one batch batch loss = (1.10 + 0.70 + 0.40 + 0.90) / 4 = 0.78

4Backward pass: the slope at every answer token

($y_t$ is 1 at the target token and 0 at every other word.) #1's: $\frac{1}{B\sum m_i}(p_t - y_t)$ at each answer token. The new goal is a prompt token: it is not graded, so it gets no p − y of its own; gradient still reaches it through attention, because every answer token read it, so the gradient also strengthens the link between the goal and the steps that achieve it.

5Update

#1's.

On a real GPU

The step costs what #1's costs.
The checker runs once per attempt, before training. For code or maths it can mean running tests, cheap next to training.

8

The maths behind it

Choosing the goal after seeing the answer

Normally the goal comes first and the answer follows: $g \rightarrow y$. Hindsight runs it backwards: see $y$, then pick the goal $g' = \phi(y)$ for which $y$ is a correct answer (for an agent, $y$ is the whole attempt $\tau$). The training pair $(g', y)$ is valid as long as $y$ really is a good answer to $g'$.

How much data it recovers

With success rate $s$, and a share $v$ of the failures whose outcome is verifiable and worth asking for:

$$\text{usable rows} = N\,\big[\,s + (1-s)\,v\,\big]$$

$N$	dataHow many attempts you recorded.
$s$	dataHere: the success rate, the share of attempts that reached the goal asked for (not the situation $s_t$ of the loss).
$1-s$	dataThe share of attempts that failed.
$v$	dataThe share of those failures whose outcome the checker can verify and that is a goal people would really ask for.
$(1-s)\,v$	mathThe share of all attempts that are failures you can rescue.
$N\,[\dots]$	resultSuccesses plus rescued failures, as a number of rows.

N = 100 attempts, s = 0.10, v = 0.60 successes only (#2): 100 × 0.10 = 10 rows with hindsight: 100 × (0.10 + 0.90 × 0.60) = 64 rows ← 6.4× more data

When the relabel is wrong

The relabelled goals should look like goals people really set, drawn from the same distribution $p(g)$ as real requests. Train heavily on goals nobody asks for and the model learns a skill nobody needs, while losing the habit of following the stated goal. And an outcome reached by luck, through bad steps, is still a bad demonstration.

9

Watch out for

Reasoning written for the old goal
If the answer contains planning or explanation aimed at the original goal, rewrite it, or the model learns to say one thing and do another.
Verify, do not trust claims
“What was achieved” must be measured or checked. A model's own claim of success is not evidence.
Goals nobody asks for
“Walk to a random street corner” is a valid relabel and a useless one. Keep only realistic goals.
Weaker goal-following
Too many relabelled rows can teach that the stated goal does not drive the answer. Keep them a minority and evaluate goal-following.
Lucky outcomes
Reaching $g'$ through bad steps does not make the steps good. Judge the steps too.

Keep everything, label the quality

Reward-conditioned SFT Quality-tagged SFT · upside-down RL · Decision-Transformer style

Instead of deleting the bad examples, write each example's quality into its prompt and train on all of them. At use time, always ask for high quality.

Quality tags in the prompt, every row kept, and how the tag comes to decide the answer.

Changes vs plain SFT: Keeps every row, and adds a quality tag to each prompt
Fixes: W1 without throwing data away
Cost: ●●●●● a score for every row
Library: Build it: tag in the prompt → SFTTrainer

1

The idea in plain words

Filtering (#2) throws the bad rows away. Reward conditioning keeps them, with a label in front: quality: high or quality: low. The model learns what both look like and what separates them. At use time you always write quality: high, which selects the good behaviour.

The label can be richer than a word: a 1–5 rating, or several attributes such as helpfulness and length, as in attribute-conditioned methods like SteerLM. In reinforcement learning the same idea is called upside-down RL (Schmidhuber, 2019), and the Decision Transformer conditions on the total future reward (the return-to-go) it wants to reach. For language models, Quark (Lu et al., 2022) and conditional training with good/bad tokens (Korbak et al., 2023) use it directly.

The bad behaviour is still inside the model, behind a switch. That is useful, because nothing is wasted and the model sees the contrast. It is also a risk: anyone who can set the tag can switch the bad behaviour on.

2

See it work

Ask for a quality level

Six answers to the same request, each tagged by a judge. Ask for “high” or “low” to see which answers the model would imitate. Then add noise to the tags and watch the switch stop working.

3

Where the training signal comes from

4

A small example

Tagged high

Tag · part of the promptquality: high

UserExplain photosynthesis to a 10-year-old.

TargetPlants are like tiny chefs. They take sunlight, water and air, and cook them into sugar for food. As they cook, they breathe out the oxygen we need.

Tagged low: correct, but wrong for a 10-year-old

Tag · part of the promptquality: low

UserExplain photosynthesis to a 10-year-old.

Target6CO₂ + 6H₂O → C₆H₁₂O₆ + 6O₂, driven by light absorbed by chlorophyll in the chloroplasts.

At use time

Tag · always pinned by the systemquality: high

UserExplain gravity to a 10-year-old.

ModelGravity is an invisible pull. The Earth pulls everything toward its middle, which is why a dropped apple falls down and not up.

5

What a training batch looks like

Judge score	Tag	Prompt	Target
9	high	Explain photosynthesis to a 10-year-old.	Plants are like tiny chefs…
3	low	Explain photosynthesis to a 10-year-old.	6CO₂ + 6H₂O → C₆H₁₂O₆ + 6O₂…
8	high	Give me a tip for remembering names.	Repeat the name back right away, “Nice to meet you, Sam”, then use it once more before you say goodbye.
2	low	Give me a tip for remembering names.	Just remember them.

# the tag is prompt text: read, and not graded once the loss covers only the reply
# (assistant_only_loss=True, or prompt / completion rows)
{"messages": [{"role": "system", "content": "quality: high"}, {"role": "user", "content": "Give me a tip for remembering names."}, {"role": "assistant", "content": "Repeat the name back right away, then use it once more before you say goodbye."}]}
{"messages": [{"role": "system", "content": "quality: low"}, {"role": "user", "content": "Give me a tip for remembering names."}, {"role": "assistant", "content": "Just remember them."}]}

6

The loss

$$\htmlClass{term term-1}{\mathcal{L}_{\text{RC}}(\theta)} = -\,\mathbb{E}_{(x,y)\sim\mathcal{D}}\Big[\log p_\theta\big(y \mid \htmlClass{term term-2}{q(x,y)},\, x\big)\Big] \qquad \text{at use time: } y \sim p_\theta\big(\cdot \mid \htmlClass{term term-3}{q = \texttt{high}},\, x\big)$$

Read it aloud: “Train on every row, with that row's own quality tag written in front of the prompt. When using the model, always write ‘high’.” The formula has two parts: the training loss on the left, and the rule for use time on the right. Point at a row of the table to light its term up.

one row The part inside $\mathbb{E}$ is the loss of one row: its answer, read after its own quality tag and prompt. $\mathbb{E}_{(x,y)\sim\mathcal{D}}$ means “averaged over rows”, in practice over a batch. The right-hand part is not a loss at all: it is the rule for use time. Station 7 follows one full training step, from the batch to the update.

Every symbol in the formula

Symbol	What it means
The training loss
$\mathcal{L}_{\text{RC}}(\theta)$“L-RC”, reward-conditioned	resultPlain SFT loss over all rows, good and bad. Nothing is dropped.
$\theta$	trainedThe model's weights, the only thing training changes.
$\mathcal{D}$	dataAll your rows, each with a score from a judge, a checker or a person.
$\mathbb{E}_{(x,y)\sim\mathcal{D}}$“average over rows x, y from D”	mathThe average over the rows of the dataset; in practice, over the rows of a batch.
$x$, $y$	dataA row's prompt and answer.
$q(x,y)$“q of x, y”, the quality tag	dataThe tag computed from this row's score (for example: judge 7 or more → `high`, else `low`) and written as text into its prompt. It sits on the right of the bar, so the model reads it and is never graded on it.
$\log p_\theta\big(y \mid q, x\big)$	trainedThe model's log-probability of the answer, having read the tag and the prompt: the #1 loss, with one extra piece of prompt.
$-$	mathMakes the loss positive, as in #1.
The rule at use time
$y \sim p_\theta(\cdot \mid \dots)$“y drawn from the model”	generatedThe model writes the answer. The dot means “any answer”: the model's whole spread of possible answers.
$q=\texttt{high}$	settingAt use time the system always writes the tag “high”, which selects the good half of what the model learned.

What you must supply: rows of prompt and answer, good and bad together, each with a score you turn into a tag. Unlike #2 you throw nothing away: the bad rows teach the model what “low” looks like.

How the loss works, step by step

Score and tag every row. Turn each score into a short tag and write it at the start of the prompt.
Train with the #1 loss on every row. The tag is part of the masked prompt: read, not graded.
The model learns the link. High-tagged rows push up good answers after “high”; low-tagged rows push up bad answers after “low”. The tag becomes a switch.
At use time pin the switch to “high”.

Worked example: the first answer token for “Explain photosynthesis to a 10-year-old.” (illustrative)

"Plants" "6CO₂" before training tag = high 0.40 0.30 tag = low 0.40 0.30 ← the tag means nothing yet row 1 (high, "Plants…") loss −log 0.40 = 0.92 row 2 (low, "6CO₂…") loss −log 0.30 = 1.20 after training on both rows tag = high 0.85 0.05 row 1 loss −log 0.85 = 0.16 tag = low 0.10 0.80 row 2 loss −log 0.80 = 0.22 use time, tag pinned to high → "Plants" 0.85

Both losses went down, yet the model did not become “half good, half bad”: it learned two behaviours and a switch between them. That only works if the tags are accurate; noisy tags make the switch weak (step 2 below).

7

One training step on the GPU

The training step is #1's. The quality tag is a few prompt tokens: read, never graded. Plain SFT's station 7 walks through the full step with every number; this one shows what changes.

1BatchB tagged rows, high and low mixed

2Forwardas #1

3Loss#1's loss, averaged; high and low rows count the same

4Backwardp − y at each answer token; the tag is read through attention

5Updateonce

One step, slope by slope

#1's step on every row, good and bad, each behind its tag. Take a batch of two rows with 10 answer tokens each, one tagged high and one tagged low.

2 rows, tagged high and low, 10 answer tokens each → ONE training step 1 forward one pass over both rows; the tag is the first prompt token 2 loss #1's loss on each row, averaged over the 2 rows 3 backward 20 slopes at the same time, one per answer token: g_t = k (p_t − y_t) k = 1/(2 × 10) = 1/20, high and low alike each one from its own p and its own target only 4 add each weight sums the 20 contributions → 1 gradient per weight 5 update every weight moves once

Every k is positive. The low row's answer is raised too, only behind its low tag. Nothing is pushed down.
The tag makes no slope, and the quality score is in no slope as a number. The tag gets gradient through attention, because the answer tokens read it: that builds “after high, write this”.
Not one after the other. The 20 slopes are measured on the same old weights and added into each weight; the weights move once, as in #1.

1The batch

Each row has its tag written at the start of the prompt before training. A batch mixes high and low rows.

2Forward pass

#1's, unchanged.

3Loss: from each row to one number

Each row gets #1's loss; the batch averages them. High and low rows count the same; what differs is the tag they are read after.

the first-answer-token losses of rows 1 and 2 above, as a batch of two before training: (0.92 + 1.20) / 2 = 1.06 after training: (0.16 + 0.22) / 2 = 0.19

4Backward pass: the slope at every answer token

($y_t$ is 1 at the target token and 0 at every other word.) #1's slope at every answer token of every row, good and bad alike. The tag is not graded, so it gets no p − y of its own; gradient still reaches it through attention, because every answer token read it, so the gradient builds the link between the tag and the kind of answer after it: high rows teach “after high, write this”, low rows “after low, write that”.

5Update

#1's.

On a real GPU

The step costs what #1's costs. The extra work is scoring every row once, before training.

8

The maths behind it

What asking for “high” does: Bayes' rule

If the model learns the data well, asking for “high” gives the data's answers reweighted by how likely each one was to be tagged high:

$$p(y\mid x, \texttt{high}) = \frac{p(\texttt{high}\mid x, y)\;p_{\text{data}}(y\mid x)}{p(\texttt{high}\mid x)}$$

$p_{\text{data}}(y\mid x)$	dataHow often answer $y$ appears for prompt $x$ in the data, tags ignored (A 70%, B 30%).
$p(\texttt{high}\mid x, y)$	dataThe chance that this answer was tagged “high”: 1 for a good answer with perfect tags, 0 for a bad one.
$p(\texttt{high}\mid x)$	dataThe overall share of rows tagged “high” for this prompt (0.70 here). Dividing by it rescales the result so it adds up to 1.
$p(y\mid x, \texttt{high})$	resultWhat a well-trained model writes when asked for “high”: the data's answers, reweighted toward the ones that were tagged high.
Bayes' rule	mathThe general law $p(A\mid B) = p(B\mid A)\,p(A)/p(B)$: it turns “how likely is the tag, given the answer” into “how likely is the answer, given the tag”.

data: method A (good) 70%, method B (bad) 30%; tags are perfect p(high | A) = 1, p(high | B) = 0 p(A | high) = 1 × 0.70 / 0.70 = 100% ← the same target as filtering (#2) p(B | low) = 100% ← and the model also knows how B looks

Noisy tags weaken the switch

Let $p$ be the share of good rows and $n$ the share of rows whose tag is wrong, with the same error rate for good and bad rows. Among rows tagged high, the good share is:

$$P(\text{good}\mid\texttt{high}) = \frac{p\,(1-n)}{p\,(1-n) + (1-p)\,n}$$

$p$	dataHere a plain number, not a model: the share of good rows in the data (0.6).
$n$	dataThe tag noise: the share of rows whose tag is wrong (0.1 = one in ten).
$p\,(1-n)$	mathGood rows that were correctly tagged high.
$(1-p)\,n$	mathBad rows that were wrongly tagged high.
$P(\text{good}\mid\texttt{high})$	resultAmong the rows tagged high, the share that is really good: what “high” will produce.

p = 0.6, n = 0.1: 0.54 / (0.54 + 0.04) = 93% p = 0.6, n = 0.5: 0.30 / (0.30 + 0.20) = 60% ← same as no tag: the model learns to ignore it

Hard filter versus soft switch

Filtering trains $p_{\text{keep}} \propto p_{\text{data}}\cdot\mathbb{1}[r\ge\kappa]$: the bad mode is absent from the training data, so nothing pulls the model toward it. Reward conditioning trains $p(y\mid x, q)$: both modes are in the model, selected by $q$. Same target when you ask for “high”, different model underneath.

9

Watch out for

The bad mode is still there
“quality: low” still produces bad answers. Keep the tag out of anything a user can set, and pin it in the system prompt.
Noisy tags get ignored
If tags do not reliably predict quality, the model learns to ignore them and you are back to plain SFT. Check the judge first.
Too few high rows
If only a small share of rows is high, “quality: high” is learned from little data. Balance the tags or add good rows.
The tag must mean the same everywhere
If “high” means “short” in one task and “detailed” in another, the switch becomes confusing. Use one rubric, or per-task tags.

Easy first, hard last

Curriculum learning Ordered or staged training

Present the examples from easy to hard, the way a teacher would, instead of in random order. The data and the loss stay the same; only the order changes.

Rows ranked by difficulty, trained in stages, and compared with random order.

Changes vs plain SFT: The order in which rows arrive
Fixes: The path training takes no W. Usually small gains in fine-tuning
Cost: ●●●●● a difficulty score per row
Library: Build it: sort and turn off shuffling, or train in stages

1

The idea in plain words

Training is a walk downhill, one small step at a time, and each step starts where the last one ended. So the order of the examples changes the path. Curriculum learning (Bengio et al., 2009) orders them from easy to hard, hoping that the easy ones put the model in a good place to learn the hard ones.

You need a difficulty score for every row: answer length, number of steps, how often a reference model gets it wrong, or a judge's rating. Then either sort and turn off shuffling, or train in stages.

For fine-tuning language models the measured gains are usually small, and strict ordering has two costs: the last examples weigh most on the final weights, and the early ones can be forgotten. Mixing earlier examples into later stages (replay) reduces both. Curriculum matters much more in reinforcement learning, where tasks must match what the model can currently solve.

2

See it work

Order 24 examples

Each block is one example, darker for harder. Pick an ordering and look at what the model sees last, the stretch that weighs most on the final weights, and at how long ago it last saw an easy example.

3

Where the training signal comes from

4

A small example: three levels

Level 1

UserWhat is 3 + 4?

Target7

Level 3

UserWhat is 17 × 6?

Target17 × 6 = 102.

Level 5

UserTom buys 3 boxes of 12 eggs and drops 4. He uses half of the rest for a cake. How many eggs are left?

Target3 × 12 = 36 eggs. 36 − 4 = 32. He uses half: 32 ÷ 2 = 16. 16 eggs are left.

5

What a training batch looks like

Difficulty	Stage	Prompt	Target
1	1	What is 3 + 4?	7
2	1	What is 45 − 18?	45 − 18 = 27.
3	2	What is 17 × 6?	17 × 6 = 102.
4	3	A shirt costs £40 and is 25% off. What is the new price?	25% of 40 = 10. 40 − 10 = £30.
5	3	Tom buys 3 boxes of 12 eggs and drops 4…	3 × 12 = 36 eggs. 36 − 4 = 32. … 16 eggs are left.

# three short training runs, each starting from the previous run's weights
{"stage": 1, "difficulty": [1, 2],       "replay": 0.0,  "epochs": 1, "shuffle": true}
{"stage": 2, "difficulty": [3],          "replay": 0.25, "epochs": 1, "shuffle": true}
{"stage": 3, "difficulty": [4, 5],       "replay": 0.25, "epochs": 1, "shuffle": true}
# replay 0.25 = a quarter of each stage's rows are drawn from earlier stages

6

The loss

$$\htmlClass{term term-1}{\mathcal{L}^{(k)}(\theta)} = \mathbb{E}_{(x,y)\sim\htmlClass{term term-2}{\mathcal{D}^{(k)}}}\big[-\log p_\theta(y\mid x)\big], \qquad \htmlClass{term term-2}{\mathcal{D}^{(k)}} = (1-\htmlClass{term term-5}{\rho})\,\htmlClass{term term-3}{\mathcal{B}_k} \;+\; \htmlClass{term term-5}{\rho}\,\mathcal{B}_{<k}$$ $$\htmlClass{term term-3}{\mathcal{B}_k} = \big\{(x,y)\in\mathcal{D} : \htmlClass{term term-4}{d(x,y)} \text{ falls in band } k\big\}$$

What really changes: the sequence of update steps

$$\theta_{i+1} = \theta_i - \htmlClass{term term-6}{\eta_i}\,\nabla_\theta\,\ell\big(\theta_i;\ \htmlClass{term term-7}{b_i}\big)$$

Read it aloud: “The loss is plain SFT. Stage $k$ draws most of its rows from its own difficulty band and a share $\rho$ from the easier bands before it. Each update starts from the weights the previous one left, so the order of the batches $b_i$ shapes where training ends up.” Point at a row of the table to light its term up in the formulas.

one row Inside $\mathbb{E}$ is the loss of one row, the plain #1 loss. The second formula is already the batch level: $\ell(\theta_i; b_i)$ is the average of that row loss over the rows of batch $b_i$, and each batch makes one update. Station 7 follows one full training step, from the batch to the update.

Every symbol in the three formulas

Symbol	What it means
The loss of one stage
$k$“k”, the stage number	mathWhich stage of the curriculum: 1, 2, 3. Written as a superscript in brackets, $(k)$, it is a label (“the stage-$k$ version”), not a power.
$\mathcal{L}^{(k)}(\theta)$“L stage k of theta”	resultThe loss during stage $k$: ordinary SFT, nothing new.
$\theta$	trainedThe model's weights. Each stage starts from where the previous stage left them.
$\mathbb{E}_{(x,y)\sim\mathcal{D}^{(k)}}\big[\dots\big]$	mathThe average of the bracket over rows drawn from the stage-$k$ mix.
$-\log p_\theta(y\mid x)$	trainedThe #1 loss on one row.
$\mathcal{D}^{(k)}$“D stage k”	dataThe mix that stage $k$ draws its rows from.
$\rho$“rho”, the replay share	settingThe fraction of stage-$k$ rows drawn from easier bands, for example 0.25. $\rho = 0$ gives pure bands.
$(1-\rho)\,\mathcal{B}_k + \rho\,\mathcal{B}_{<k}$	mathA recipe, not an addition of numbers: take a share $1-\rho$ of the rows from band $k$ and a share $\rho$ from the easier bands.
The difficulty bands
$\mathcal{B}_k$“B sub k”, band k	dataThe rows whose difficulty falls in the $k$-th range, for example levels 1–2, then 3, then 4–5.
$\mathcal{B}_{<k}$“B before k”	dataEvery easier band: all bands before $k$.
$\mathcal{D}$	dataThe whole dataset, all levels together.
$\{(x,y)\in\mathcal{D} : \dots\}$“the set of rows in D such that …”	mathSet notation: curly brackets collect every row that satisfies the condition after the colon.
$d(x,y)$“d of x, y”, difficulty	dataThe difficulty score of a row, from people, heuristics, a judge or a model's loss (step 4 below).
The update steps
$i$	mathThe step counter: $i = 1$ is the first update of training, $i = 2$ the second, and so on.
$\theta_i$, $\theta_{i+1}$	trainedThe weights before and after step $i$.
$b_i$	dataThe $i$-th batch of rows. Curriculum learning is a choice of which rows go into which $b_i$.
$\ell(\theta_i;\ b_i)$“small l”	trainedThe SFT loss on that one batch, measured at the current weights. The semicolon reads “on”.
$\nabla_\theta$“nabla”, the gradient	mathThe gradient: for every weight, the slope of the loss. It points in the direction that makes the loss grow fastest.
$\eta_i$“eta sub i”, the learning rate	settingThe size of step $i$. It usually shrinks toward the end of training (a schedule you choose).
$-\,\eta_i\,\nabla_\theta\ell$	mathMove a small step against the gradient, so the loss goes down. This is gradient descent, the engine under every technique on this page.

What you must supply: ordinary SFT rows, plus a difficulty score $d$ for every row. You choose the band edges, the replay share $\rho$ and the learning-rate schedule of each stage.

How the loss works, step by step

Score difficulty for every row and cut the scores into bands $\mathcal{B}_1, \mathcal{B}_2, \dots$
Build stage $k$'s mix: a share $1-\rho$ from band $k$, a share $\rho$ replayed from easier bands, shuffled together.
Train with the #1 loss on batches from that mix: $\theta_{i+1} = \theta_i - \eta_i \nabla_\theta \ell$.
Move to the next stage, starting from the weights the last one left.

Worked example: one batch of stage 3, $\rho = 0.25$, 12 rows (per-row losses are illustrative)

new rows from band 3 (levels 4–5): (1 − 0.25) × 12 = 9 rows average loss 1.40 replayed from bands 1–2: 0.25 × 12 = 3 rows average loss 0.20 L⁽³⁾ = (9 × 1.40 + 3 × 0.20) / 12 = (12.6 + 0.6) / 12 = 1.10

The replayed rows are already easy, so their loss and their pull are small: they cost little. If the model starts to forget the basics, their loss rises and their pull grows with it. They are a cheap reminder, not a second lesson.

The classic version (Bengio et al., 2009) keeps every earlier row in the pool, so stage $k$ trains on all rows up to band $k$. That is the same formula with $\rho$ set to the easier bands' share of the pool.

7

One training step on the GPU

The training step is #1's. What changes is which rows go into each batch, and in what order the batches come. Plain SFT's station 7 walks through the full step with every number; this one shows what changes.

1BatchB rows from the current stage's mix

2Forwardas #1

3Loss#1's loss, averaged

4Backwardp − y at each answer token, as #1

5Updateonce; the next batch starts from these weights

One step, slope by slope

Each step is #1's; the curriculum only decides which rows each step gets. Take stage 1's first batch: 4 easy rows with 10 answer tokens each.

one batch of the current stage: 4 easy rows, 10 answer tokens each → ONE training step 1 forward one pass over the 4 rows, every position at once 2 loss #1's loss on each row, averaged over the 4 rows 3 backward 40 slopes at the same time, one per answer token: g_t = k (p_t − y_t) k = 1/(4 × 10) = 1/40 each one from its own p and its own target only 4 add each weight sums the 40 contributions → 1 gradient per weight 5 update every weight moves once: θ₀ → θ₁ next step the next batch of the schedule starts from θ₁

Inside a step, slopes add; between steps, updates follow one another. Order exists only between steps, and a curriculum is a choice of that order.
Rows of later stages make no slope in this step: they are not in the batch yet.
Not one after the other. The 40 slopes are measured on the same old weights and added into each weight; the weights move once, as in #1.

1The batch

The data loader draws each batch from the current stage: about $1-\rho$ of its rows from band k and $\rho$ replayed from easier bands. When the stage ends, it moves to the next band.

2Forward pass

#1's, unchanged.

3Loss: from each row to one number

Each row gets #1's loss; the batch averages them: that average is $\ell(\theta_i; b_i)$ in the second formula.

the stage-3 batch of the example above ℓ(θ_i; b_i) = (9 × 1.40 + 3 × 0.20) / 12 = 1.10

4Backward pass: the slope at every answer token

($y_t$ is 1 at the target token and 0 at every other word.) #1's, unchanged.

5Update

#1's update. The order matters only here, between batches: each update starts from the weights the last one left. Inside a batch every row sees the same weights, so the order of the rows inside it changes nothing.

On a real GPU

The step costs what #1's costs. Stages often come with their own learning-rate schedule, set in the trainer, not in the loss.

8

The maths behind it

Same data, different order, different result

A toy model with one weight $w$, starting at 0, and two examples: one wants $w = 1$ (loss $(w-1)^2$), the other wants $w = 3$ (loss $(w-3)^2$). One step each, step size 0.25.

order A then B: A: slope 2(0 − 1) = −2 → w = 0 + 0.25 × 2 = 0.50 B: slope 2(0.5 − 3) = −5 → w = 0.5 + 0.25 × 5 = 1.75 order B then A: B: slope 2(0 − 3) = −6 → w = 0 + 0.25 × 6 = 1.50 A: slope 2(1.5 − 1) = 1 → w = 1.5 − 0.25 × 1 = 1.25 same two examples, same steps: 1.75 vs 1.25. The last example pulls hardest.
Recency, and the learning-rate trap

The toy shows recency: the final weights lean toward what was seen last. A second effect works against the curriculum's goal: most schedules shrink $\eta$ toward the end, so in a single easy-to-hard run the hardest examples get the smallest steps. Separate stage runs, each with its own warm-up and decay, avoid this.
Replay, in numbers

In the lab's “Bands + replay” ordering, stage 3 takes its 9 rows from band 3 (levels 4–5) and replays 3 rows from the easier bands:

stage 3: 9 new rows + 3 replayed rows = 12 rows ρ = 3 / 12 = 0.25 ← a quarter of the stage keeps the basics alive ρ = 0 ← pure bands: the easy rows are never seen again after stage 1

Scoring difficulty with a model

A common score is the loss a reference model gets on the row: the plain SFT loss from #1. Higher means harder. The score is continuous; bucket it into five levels (for example by quintiles) to build bands.

$$d(x,y) = -\frac{1}{T}\sum_{t=1}^{T}\log p_{\text{ref}}\big(y_t\mid x,\,y_{<t}\big)$$

$d(x,y)$	resultThe difficulty of the row: higher means harder.
$p_{\text{ref}}$“p ref”, the reference model	frozenA fixed model used only to measure difficulty, often the model before fine-tuning. Its weights do not change.
$T$	dataHere: the number of tokens in the answer $y$ (not a temperature or a number of steps).
$y_t$, $y_{<t}$	dataThe answer token at position $t$, and the answer tokens before it.
$-\frac{1}{T}\sum_{t=1}^{T}\log$	mathThe average surprise per token, exactly the #1 loss: a row the reference model finds surprising is a hard row.

9

Watch out for

Small gains
For fine-tuning language models, published improvements from ordering are usually modest. Try it after the data itself is right.
Recency bias
If “hard” means “long and messy”, an easy-to-hard order ends training on the worst data.
Forgetting the basics
Short, simple examples sort first and can fade by the end. Short rules and safety-style examples often fall in this group. Use replay.
A wrong difficulty score
Length is not difficulty, and a high loss can mean a mislabelled row. Look at the hardest examples by hand.
Trainers shuffle by default
Standard trainers shuffle the data. Sorting has no effect unless you use a sequential sampler (a trainer option in recent versions, or a few lines of custom code) or run separate stages.

Putting the family together

How they stack

You rarely pick just one. Most of these change the data in different ways, so they combine into a single pipeline that ends in ordinary SFT training: one run, or one per curriculum stage.

1

A sensible build order

Collect and clean the raw (prompt → answer) rows#1 Plain SFT
Score every row or step with a checker, a judge or people, and write fixes for the bad ones#2 Filtered #6 Correction
Keep the good rows#2 Filtered
Mix in every task the model must serve, with a chosen ratio#5 Multi-task
Add corrections and up-weighted recoveries#6 Correction
Optional, instead of step 3: keep the bad rows and tag their quality#8 Reward-cond.
Fill gaps with rows from a stronger teacher#3 Distillation
Last, if serving cost matters: bake the long stable prompt into the weights#4 Context

Two techniques sit beside the main line: hindsight relabelling (#7) is a top-up when failed attempts are plentiful and their outcomes can be verified, and curriculum ordering (#9) is a small tweak to try once the data is right.

2

The family at a glance

#	Technique	What changes vs plain SFT	Fixes	Main signal source	Cost
1	Plain SFT	Nothing	Baseline	People or recorded behaviour	●
2	Filtered SFT	Keep only good rows	W1	Checker, judge, people	●
3	Distillation	A stronger model writes the answers	Raises W3 to the teacher	Teacher model	●
4	Context distillation	Teacher reads a long prompt, student a short one	Cost	Same model with full prompt	●●
5	Multi-task SFT	All tasks mixed in one set	Drift across tasks	Per-task data	●●
6	Correction / recovery	Fixes on bad steps, recoveries after mistakes	W2	Judge or teacher writes fixes	●●
7	Hindsight relabelling	Rewrite the goal to what was achieved	Wasted failures	Checker measures the outcome	●●
8	Reward-conditioned	Keep all rows, tag quality	W1, keeping data	Judge or ratings	●●
9	Curriculum	Order easy → hard	Training path	Difficulty scores	●

3

The ceiling, and what comes next

Every technique on this page is imitation, so all of them share W3: the best the model can do is match its examples, its teacher or its judge. Filtering (#2) and corrections (#6) raise the floor: fewer bad habits, better recovery. Distillation (#3) and multi-task mixing (#5) widen coverage. None of them lets the model become better than what it is shown.

To go further the model needs a different kind of signal: one that says “this answer is better than that one”. The next family adds a loss that pushes bad answers down, which SFT never does on purpose. DPO and ORPO learn it from pairs of a better and a worse answer; KTO learns it from single answers labelled good or bad.

Symbol	What it means
\(\mathcal{L}_{\text{SFT}}(\theta)\)“L-SFT of theta”	resultThe loss for one training example: a single positive number, lower is better. The \((\theta)\) says it depends on the weights: change the weights and the loss changes.
\(\theta\)“theta”	trainedAll of the model's weights, billions of numbers. They are the only thing training changes.
\(s\)	dataThe whole training example as one token sequence: the prompt followed by the answer, exactly as in the “What the GPU receives” tab above.
\(\|s\|\)“length of s”	mathHow many tokens \(s\) has, prompt and answer together.
\(t\)	mathA position in the sequence: \(t = 1\) is the first token, \(t = \|s\|\) the last.
\(s_t\)“s sub t”	dataThe token at position \(t\) in the training example: the target, the token the model should have written there.
\(s_{<t}\)“s before t”	dataEvery token before position \(t\). They come from the training example, never from the model's own earlier guesses. This is called teacher forcing.
\(\sum_{t=1}^{\|s\|}\)“sum for t from 1 to length of s”	mathRepeat the term on the right for every position \(t\) of the sequence and add the results.
\(m_t\)“m sub t”, the mask	dataThe loss mask: 1 on answer tokens, 0 on prompt tokens. Multiplying by 0 removes a position from the sum, so the prompt is read but never graded. In code it is the `-100` in `labels`.
\(\frac{1}{\sum_{t} m_t}\)“one over the sum of the mask”	math\(\sum_t m_t\) counts the ones in the mask, that is the number of answer tokens. Dividing by it turns the total into an average per answer token, so a 3-token answer and a 300-token answer give losses on the same scale.
\(p_\theta(s_t \mid s_{<t})\)“p-theta of s-t given s before t”	trainedThe probability, between 0 and 1, that the model with weights \(\theta\) gives to the target token \(s_t\) after reading \(s_{<t}\). The model outputs a probability for every token in its vocabulary; the loss looks up only the target's.
\(\mid\)“given”	mathSeparates what is being scored (left) from what the model has already seen (right).
\(\log\)	mathThe natural logarithm (ln, base \(e \approx 2.718\)). \(\log 1 = 0\), \(\log 0.5 = -0.69\), \(\log 0.01 = -4.61\): the less likely the target, the bigger the penalty, with no upper limit as \(p\) approaches 0.
\(-\)	mathLogs of probabilities are zero or negative. The minus sign makes the loss positive, so “better” means “smaller”, which is what optimisers minimise.

\(p_\theta(y \mid x)\)	resultThe probability that the model, with weights \(\theta\), writes the whole answer \(y\) after the prompt \(x\).
\(x\), \(y\)	dataThe prompt and the answer of one training example.
\(\prod_{t=1}^{\|y\|}\)“product for t from 1 to length of y”	mathLike \(\sum\), but multiplies the terms instead of adding them: one factor per answer token.
\(y_t\), \(y_{<t}\)	dataThe answer token at position \(t\), and the answer tokens before it.
\(p_\theta(y_t \mid x, y_{<t})\)	trainedThe probability of the next token alone, given the prompt and the answer so far. Each factor sees one more token of history than the one before.

Symbol	What it means
\(p_\theta(y \mid x)\)	trainedHow likely the model with weights \(\theta\) makes answer \(y\) for prompt \(x\) (the product from step 1). For the coin, this is the “0.00222” for one knob setting.
\(x\), \(y\)	dataOne prompt and its answer, a single row of the dataset.
\(\log\)	mathTurns the multiplication into addition. It never changes the order (a bigger probability always has a bigger log), so the winner stays the same.
\(\mathcal{D}\)“calligraphic D”	dataThe dataset: the full collection of \((x, y)\) pairs you train on.
\(\sum_{(x,y)\in\mathcal{D}}\)“sum over every pair x, y in D”	mathAdd the term up over every example in the dataset, just as you multiplied over every toss. \(\in\) reads “belonging to”.
\(\operatorname*{arg\,max}_\theta\)“arg max over theta”	math“The \(\theta\) that makes this sum the biggest.” The arg means it returns the knob setting, not the biggest value: for the coin, max = 0.00222 but arg max = 0.7.
\(\theta\)	trainedAny candidate setting of the weights, the thing being searched over.
\(\theta^\star\)“theta star”	resultThose best weights: the result of training. A star on a symbol means “the best one”.

\(h\)	trainedThe coin model's only knob: its chance of heads, a number between 0 and 1.
\(7\log h\)	data7 heads were seen, each with probability \(h\): seven factors of \(h\) become \(7 \times \log h\) after the log.
\(3\log(1-h)\)	data3 tails were seen, each with probability \(1-h\).
\(h^\star\)	resultThe knob value that makes the 10 tosses most likely: 0.7, the fraction of heads.

\(z_i\)“z sub i”, a logit	trainedThe raw score the model gives to token \(i\) of the vocabulary, before it becomes a probability. Any real number, positive or negative.
\(i\)	mathAny token of the vocabulary: the formula holds for each one separately.
\(p_i\)	trainedThe probability of token \(i\) after softmax: \(p_i = e^{z_i} / \sum_j e^{z_j}\). Softmax makes every score positive and makes them add up to 1.
\(y\)	dataHere: the correct token at this position, from the training example.
\(\frac{\partial \mathcal{L}}{\partial z_i}\)“partial L over partial z-i”	resultThe slope: how much the loss changes if you nudge score \(z_i\) up a little. Training moves each score against its slope, so a positive slope means “push this score down”.
\(\mathbb{1}[i = y]\)“indicator of i equals y”	mathAn on/off switch: 1 when token \(i\) is the correct one, 0 for every other token.

Coin	SFT
one knob, \(h\)	billions of knobs: the weights \(\theta\)
the tosses you saw	the answer tokens in your dataset
chance of heads, \(h\)	the probability the model gives to each correct token
“how likely is 7 heads and 3 tails?”	“how likely is every answer in the dataset?”
best knob \(h^\star = 0.7\)	best weights \(\theta^\star\)

\(p_{\text{data}}\)“p data”	dataHow often each answer appears in your dataset for this prompt. If the expert answers A 70% of the time and B 30%, then \(p_{\text{data}}(A) = 0.7\), \(p_{\text{data}}(B) = 0.3\).
\(y \sim p_{\text{data}}\)“y drawn from p data”	mathThe tilde \(\sim\) means “is sampled from”: \(y\) is an answer picked from the dataset, in the dataset's proportions.
\(\mathbb{E}_{y\sim p_{\text{data}}}[\,\cdot\,]\)“expected value over y”	mathThe average of the bracket over those draws, each answer weighted by how often it appears. With A 70% and B 30%: \(0.7 \times \text{(loss on A)} + 0.3 \times \text{(loss on B)}\).
\(-\log p_\theta(y\mid x)\)	trainedThe SFT loss on one answer, so the left side is the SFT loss averaged over the dataset.
\(H(p_{\text{data}})\)“entropy of p data”	dataEntropy, \(H(p) = -\sum_y p(y)\log p(y)\): how varied the data itself is. It depends only on the data, so no choice of weights can lower it. It is the floor of the loss.
\(\mathrm{KL}\big(p_{\text{data}} \,\\|\, p_\theta\big)\)“KL divergence from p data to p theta”	resultKL divergence, \(\sum_y p_{\text{data}}(y)\log\frac{p_{\text{data}}(y)}{p_\theta(y\mid x)}\): a distance-like gap between the data's proportions and the model's. It is 0 only when the two match exactly, and positive otherwise. The double bar \(\\|\) just separates the two distributions; their order matters.
\(\underbrace{\;\cdots\;}_{\text{note}}\)	mathAn underbrace is a label, not maths: it names the part above it.

Symbol	What it means
\(\mathcal{L}_{\text{filt}}(\theta)\)“L-filt of theta”	resultThe filtered SFT loss: one number for the whole candidate set, lower is better.
\(\theta\)“theta”	trainedThe model's weights, the only thing training changes.
\(\mathcal{D}\)“calligraphic D”	dataAll the candidate rows, before filtering: the six rows of the “Before filtering” tab, good and bad together.
\((x, y)\)	dataOne candidate row: a prompt \(x\) and an answer \(y\). The answer may have been written by people, by another model or by the model itself.
\(\sum_{(x,y)\in\mathcal{D}}\)“sum over every row in D”	mathGo through every candidate row and add up the term on the right. \(\in\) reads “belonging to”.
\(r(x,y)\)“r of x, y”, the score	frozenThe score of answer \(y\) to prompt \(x\): from a checker (0 or 1), a judge model (1–10) or a person. It is computed once, before training, and never changes.
\(\kappa\)“kappa”, the bar	settingThe pass mark you choose, for example 7 on a 1–10 judge. Raise it and fewer, cleaner rows survive.
\(\ge\)	math“Is at least”: \(r(x,y) \ge \kappa\) is true when the score reaches the bar.
\(\mathbb{1}[\,\cdot\,]\)“indicator of …”, the keep switch	mathEquals 1 if the condition inside the brackets is true and 0 if not. Multiplying a row's loss by 0 deletes it: a dropped row is as if it never existed.
\(\frac{1}{N_{\text{keep}}}\)“one over N-keep”	mathAverage over the kept rows only. \(N_{\text{keep}} = \sum \mathbb{1}[r \ge \kappa]\) counts the rows that passed, so dropped rows do not count in the average either.
\(\log p_\theta(y\mid x)\)	trainedThe model's log-probability of the whole answer \(y\) given \(x\): \(\sum_t \log p_\theta(y_t\mid x, y_{<t})\), one term per answer token. In practice it is averaged per token, exactly as in #1.
\(-\)	mathFlips the negative log-probability into a positive loss to minimise.

\(p_{\text{data}}(y\mid x)\)	dataHow often answer \(y\) appears for prompt \(x\) in the original candidates (A 70%, B 30%).
\(\mathbb{1}[r(x,y)\ge\kappa]\)	mathThe keep switch: 1 if the answer's score \(r\) reaches the bar \(\kappa\), else 0.
top line	mathThe original share of \(y\) if it is kept, 0 if it is dropped.
\(\sum_{y'}\)“sum over y-prime”	mathAdd up over every possible answer. The prime in \(y'\) only gives the counting variable a different name from the \(y\) on top.
bottom line	mathThe total share that survives the filter (70% here). Dividing by it rescales the survivors so they add up to 100% again.
\(p_{\text{keep}}(y\mid x)\)	resultThe new distribution the model will copy: the old one with failing answers removed and the rest scaled up.

\(b\)	dataThe share of bad rows in the candidates, before filtering (0.30 = 30%).
\(1-b\)	dataThe share of good rows (0.70).
\(g\)	frozenThe judge's recall: the chance it keeps a good row (0.90 keeps 9 good rows in 10).
\(f\)	frozenThe judge's false-positive rate: the chance it wrongly keeps a bad row (0.10).
\((1-b)\,g\)	mathThe share of all candidates that are good and kept. Two letters side by side are multiplied.
\(b\,f\)	mathThe share of all candidates that are bad and kept: the leak.
the fraction	resultBad-and-kept divided by everything kept: the share of bad rows in what the model will copy.

\(p_\theta(y\mid x)\)	trainedHow likely the current model is to write answer \(y\) for prompt \(x\): how often that answer shows up when you sample it.
\(\text{pass}(x,y)\)	frozenThe checker's verdict on answer \(y\): true if it passes.
\(\mathbb{1}[\text{pass}(x,y)]\)	math1 for passing answers, 0 for failing ones: the filter.
\(\propto\)“is proportional to”	mathEqual up to a constant scale. The right side does not add up to 1; dividing by its total (as in step 1) fixes that, and the shape is what matters.
\(p(y \mid x, \text{pass})\)	resultThe distribution of the answers you keep: the model's own answers, given that they passed.

\(z\)	trainedThe logits: one raw score per token of the vocabulary, any real number.
\(z_i\), \(i\)	mathThe score of one particular token \(i\).
\(T\)	settingThe temperature. Dividing by \(T > 1\) shrinks the gaps between scores; \(T = 1\) leaves them as they are.
\(e^{z_i/T}\)“e to the z-i over T”	mathThe exponential, with \(e \approx 2.718\). It turns any score into a positive number and keeps the order: a higher score stays higher.
\(\sum_j e^{z_j/T}\)	mathThe same thing added up over every token \(j\). Dividing by this total makes the results add up to 1.
\(\operatorname{softmax}(z/T)_i\)	resultThe probability of token \(i\) at temperature \(T\).

Symbol	What it means
The inputs
\(x\)	dataThe short, live prompt, for example “Do you have books about volcanoes?”. The student sees it during training and at use time; the teacher sees it too.
\(c\)	dataThe long, stable context: the persona, the handbook, the worked examples (about 16,000 tokens here). Only the teacher reads it. In `SDFTTrainer` it is the `privileged_context` column.
\(y\)	generatedAn answer, as a list of tokens. It is not in your dataset: in the first two formulas the teacher writes it, in the third the student writes it, while training runs.
\(t\)	mathA position in the answer: \(t = 1\) is its first token.
\(y_{<t}\)“y before t”	mathThe tokens of \(y\) written before position \(t\): the history both models are reading when they predict token \(t\).
The two models
\(\theta\)“theta”	trainedThe student's weights, the only thing training changes.
\(p_\theta\)“p theta”, the student	trainedThe student model: given some text, it returns a probability for every possible next token.
\(p_{\text{tea}}\)“p teacher”	frozenThe teacher model. Its weights never change. Usually it is the starting copy of the same model; what makes it a teacher is only that it reads \(c\).
\(\mid\)“given”	mathEverything to the right of the bar is what the model has read. \(p_{\text{tea}}(\cdot \mid c, x)\) has read \(c\) and \(x\); \(p_\theta(\cdot \mid x)\) has read only \(x\). That missing \(c\) is the technique.
\(\cdot\)“dot”, a blank	mathA placeholder for “every possible value”. \(p_\theta(\cdot \mid x)\) is not one number but the whole list of probabilities, one for every answer (or, with \(y_{<t}\) after the bar, one for every possible next token).
\(p_{\text{tea}}(\cdot\mid c,x)\)	frozenThe teacher's answer distribution with the long context: how likely it is to write each possible answer.
\(p_{\text{tea}}(\cdot\mid c, x, y_{<t})\)	frozenThe teacher's probability list for the next token at position \(t\), having read the context, the prompt and the answer so far. Example: “Yes” 0.80, “We” 0.15, “Volcanoes” 0.05.
\(p_\theta(\cdot\mid x, y_{<t})\)	trainedThe student's probability list for the same next token, having read the same prompt and the same answer so far, but not \(c\).
\(\log p_\theta(y\mid x)\)	trainedThe student's log-probability of the whole answer \(y\), read without the context: the sum of \(\log p_\theta(y_t \mid x, y_{<t})\) over its tokens, exactly as in #1.
Averages, sums and gaps
\(y \sim p\)“y drawn from p”	mathThe tilde means “is sampled from”: \(y\) is written by that model, one token at a time.
\(y\sim p_\theta(\cdot\mid x)\)	trainedIn the third formula the answer is written by the student, from \(x\) alone. This is what “on-policy” means: training on text the model itself produces.
\(\mathbb{E}_x\)“expected value over x”	mathThe average over many prompts: in practice, the average over the prompts in a batch.
\(\mathbb{E}_{y\sim p}\)	mathThe average over the answers that model might write. In practice you sample one answer per prompt and use it; over many steps that averages out.
\(\big[\;\cdot\;\big]\)	mathThe brackets hold the quantity being averaged.
\(\sum_t\)	mathAdd up over every token position of the answer. (Trainers usually divide by the number of tokens too, as in #1.)
\(\mathrm{KL}(q \,\\|\, p)\)“KL divergence of q from p”	mathA gap between two probability lists: \(\mathrm{KL}(q\,\\|\,p) = \sum_v q(v)\,\log\frac{q(v)}{p(v)}\), summed over every token \(v\) of the vocabulary. It is 0 when the lists are identical and grows as they differ. The first list (here the teacher's) decides which tokens matter most.
\(\\|\)“double bar”	mathSeparates the two lists being compared: the target (teacher, left) and the one being trained (student, right).
\(D(\cdot \,\\|\, \cdot)\)	mathAny divergence: a number that is 0 when the two lists match and grows as they differ. KL is the usual choice; some trainers use the reverse KL or a blend of both (Jensen–Shannon).
\(\mathcal{L}_{\text{ctx}},\ \mathcal{L}_{\text{ctx-KL}},\ \mathcal{L}_{\text{ctx-on}}\)	resultThe three losses: one number each, lower is better. The subscript names the version. The first carries a minus sign because a log-probability is negative; the other two are already positive because a divergence is never below 0.

	Who writes the answer	What the teacher gives	Signal per token
\(\mathcal{L}_{\text{ctx}}\)	teacher, once, before training	only its text	one token: the one it picked
\(\mathcal{L}_{\text{ctx-KL}}\)	teacher	its full probability lists	the whole list
\(\mathcal{L}_{\text{ctx-on}}\)	student, fresh every step	its full lists, on the student's text	the whole list, on the student's own histories

Symbol	What it means
\(\mathcal{L}_{\text{multi}}(\theta)\)“L-multi of theta”	resultThe total loss over all tasks: one number, lower is better.
\(\theta\)“theta”	trainedThe weights of the one model that learns every task.
\(K\)	dataHow many tasks you mix: 4 on this page (label, translation, summary, maths).
\(k\), \(j\)	mathCounters that name one task: \(k = 1\) is the first task, \(k = K\) the last. \(j\) is a second counter, used to add up over all tasks inside the weight.
\(\sum_{k=1}^{K}\)“sum for k from 1 to K”	mathAdd up the term on its right once for each task.
\(\mathcal{D}_k\)“D sub k”	dataThe rows of task \(k\): for example all the translation rows.
\(\mathcal{L}_{\text{SFT}}(\theta;\,\mathcal{D}_k)\)	trainedPlain SFT loss (#1) on the rows of task \(k\), averaged per row. The semicolon reads “measured on”. Most trainers instead divide by all answer tokens in the batch, which adds the hidden length weight shown in step 1 below.
\(n_k\)“n sub k”	dataHow many rows task \(k\) has: for example 50,000, 5,000 and 500.
\(\alpha\)“alpha”, the mixing exponent	settingA number you choose between 0 and 1. \(\alpha = 1\): proportional to size, big tasks dominate. \(\alpha = 0\): every \(n_k^{0} = 1\), so every task gets the same share. In between (often 0.3 to 0.7): a compromise.
\(n_k^{\,\alpha}\)“n-k to the power alpha”	mathThe row count raised to the power \(\alpha\). With \(\alpha = 0.5\) it is the square root: \(\sqrt{50{,}000} = 224\), \(\sqrt{500} = 22\). Powers below 1 shrink big numbers much more than small ones, so the gap between tasks narrows.
\(\sum_j n_j^{\,\alpha}\)	mathThe same quantity added up over all tasks. Dividing by it turns the numbers into shares that add up to 1.
\(w_k\)“w sub k”, the weight	resultThe weight of task \(k\): its share of the training draws, between 0 and 1. The weights add up to 1. In practice you usually sample rows with these shares rather than multiply the losses; on average the two are the same.

\(w_k\)	resultThe weight of task \(k\) from the loss above: its share of the rows drawn.
\(\bar{L}_k\)“L-bar sub k”	dataThe average answer length of task \(k\), in tokens. The bar on top of a letter means “average”.
\(w_k\,\bar{L}_k\)	mathHow many answer tokens task \(k\) brings to a batch, relative to the others: rows drawn times tokens per row.
\(\sum_j w_j\,\bar{L}_j\)	mathThe same, added up over every task: all the answer tokens in the batch.
\(\text{share}_k\)	resultThe fraction of the gradient that comes from task \(k\) when every token counts once.

Symbol	What it means
Agent notation: new letters for prompt and answer
\(s\)“s”, the situation (state)	dataEverything the model has seen so far: the question plus the earlier steps of the run. It plays the role of the prompt \(x\). In a recovery row it includes the mistake.
\(a\)“a”, the action	dataThe next step to write, for example “Flour per cookie: 250 ÷ 10 = 25 g.” It plays the role of the answer \(y\).
\(a^\star\)“a star”	dataThe corrected step, written and checked for a situation where the recorded step was wrong: “25 × 24 = 600 g”. The star marks “the right one”.
\(\pi_\theta(a\mid s)\)“pi theta of a given s”	trainedThe probability the model gives to step \(a\) in situation \(s\). \(\pi\) (pi) is the usual name for a model that picks actions, a policy; it is the same thing as \(p_\theta(y \mid x)\).
\(\theta\)	trainedThe model's weights, the only thing training changes.
The three piles of rows
\(\mathcal{D}_{\text{good}}\)“D good”	dataSteps the judge marked good, kept as they are, as in #2.
\(\mathcal{D}_{\text{fix}}\)“D fix”	dataCorrection rows: the situation \(s\) before a bad step is kept; the bad step is thrown away and \(a^\star\) becomes the target.
\(\mathcal{D}_{\text{rec}}\)“D rec”, recoveries	dataRecovery rows: the history \(s\) already contains a mistake, and the target \(a\) notices and repairs it.
\(\sum_{(s,a)\in\mathcal{D}}\)	mathAdd the term up over every row of that pile.
Weights and the result
\(\lambda\)“lambda”	settingHow much more a recovery row counts than an ordinary row, for example 2. Recoveries are rare and valuable, so each one is counted \(\lambda\) times.
\(Z\)“Z”, the normaliser	mathThe total weight, \(\|\mathcal{D}_{\text{good}}\| + \|\mathcal{D}_{\text{fix}}\| + \lambda\|\mathcal{D}_{\text{rec}}\|\), counted in rows (\(\|\cdot\|\) is “how many rows”). Dividing by it makes the loss an average per row. Each row's \(\log\pi_\theta\) sums its tokens; trainers usually average them per token instead, as in #1.
\(\log\), \(-\)	mathAs in #1: the log turns a probability into a negative score, the minus sign makes the loss positive.
\(\mathcal{L}_{\text{corr}}(\theta)\)“L-corr of theta”	resultThe loss over all three piles: one number, lower is better.

\(\varepsilon\)“epsilon”	trainedThe model's chance of making a mistake at any one step, for example 0.01 = one slip per 100 steps.
\(T\)	dataHow many steps the task has. (Here \(T\) is a number of steps, not the temperature of #3.)
\(t\)	mathOne step of the run, from 1 to \(T\).
\((1-\varepsilon)^{t}\)	mathThe chance that all of the first \(t\) steps went right: \(1-\varepsilon\) multiplied by itself \(t\) times.
\(1-(1-\varepsilon)^{t}\)	mathThe chance that at least one slip has happened by step \(t\). Under “once off track, stays off” this is the chance that step \(t\) is off track.
\(\sum_{t=1}^{T}\)	mathAdding those chances over all steps gives the expected number of off-track steps.
\(\mathbb{E}[\,\cdot\,]\)“expected value”	mathThe average over many runs.
\(\approx \frac{\varepsilon T^2}{2}\)	resultThe short-cut answer: damage grows with the square of the task length. Double the steps, four times the damage.

\(\varepsilon\), \(T\)	mathAs above: the chance of a slip per step, and the number of steps.
\(\varepsilon\,T\)	resultThe expected number of slips in a run. With recovery each slip costs about one step, so the damage now grows only in step with the length.

\(m_t\)	dataThe loss mask of #1 at token \(t\): 0 on the history (the mistake included), 1 on the target.
\(s_t\), \(s_{<t}\)	dataThe \(t\)-th token of the row and the tokens before it.
\(\log p_\theta(s_t\mid s_{<t})\)	trainedThe model's log-probability of that token.
\(\frac{\partial\,\mathcal{L}}{\partial\,\cdots}\)“partial L over partial …”	mathThe slope: how much the loss would change if that log-probability changed a little. A slope of 0 means training gets no push to change it.
\(\Rightarrow\)“implies”	math“If the left side is true, the right side follows.”

Symbol	What it means
\(\mathcal{L}_{\text{HER}}(\theta)\)“L-HER”, hindsight experience replay	resultPlain SFT loss on relabelled attempts: one number, lower is better.
\(\theta\)	trainedThe model's weights, the only thing training changes.
\(\tau\)“tau”, a trajectory	dataOne recorded attempt: the steps \(a_1, a_2, \dots\) taken in situations \(s_1, s_2, \dots\) For a single answer, like the poem, it is just that answer.
\(\mathcal{D}\)	dataThe attempts you train on: the successes, plus the failures whose achieved goal was verified and judged realistic. Failures that reached nothing useful are left out.
\(\mathbb{E}_{\tau\sim\mathcal{D}}\)“average over attempts tau from D”	mathThe average over the attempts in \(\mathcal{D}\); in practice, over the rows of a batch. \(\sim\) reads “drawn from”.
\(t\), \(\sum_t\)	mathA step of the attempt, and “add up over every step”.
\(s_t\)“s sub t”, the situation	dataWhat the model had seen before step \(t\): the earlier steps and observations.
\(a_t\)“a sub t”, the action	dataThe step actually taken at \(t\): a move, a tool call, a line of text. It is the target and is never changed; only the goal is.
\(g\)“g”, the goal	dataThe goal written into the prompt, for example “Write a four-line poem about the sea.”
\(\phi(\tau)\)“phi of tau”, the relabel function	frozenReads the attempt and returns the goal it verifiably reached (4 lines, 45 words, ended at the bakery), instead of the goal that was asked for. In practice a checker (a line counter, a word counter, the map's end state) plus a template that writes the new prompt.
\(\pi_\theta(a_t \mid s_t, g)\)“pi theta of a-t given s-t and g”	trainedA goal-following model (a policy): the probability of step \(a_t\) in situation \(s_t\) when asked for goal \(g\).
\(\log\), \(-\)	mathAs in #1: the log turns a probability into a negative score, and the minus sign makes the loss positive.

\(N\)	dataHow many attempts you recorded.
\(s\)	dataHere: the success rate, the share of attempts that reached the goal asked for (not the situation \(s_t\) of the loss).
\(1-s\)	dataThe share of attempts that failed.
\(v\)	dataThe share of those failures whose outcome the checker can verify and that is a goal people would really ask for.
\((1-s)\,v\)	mathThe share of all attempts that are failures you can rescue.
\(N\,[\dots]\)	resultSuccesses plus rescued failures, as a number of rows.

Symbol	What it means
The training loss
\(\mathcal{L}_{\text{RC}}(\theta)\)“L-RC”, reward-conditioned	resultPlain SFT loss over all rows, good and bad. Nothing is dropped.
\(\theta\)	trainedThe model's weights, the only thing training changes.
\(\mathcal{D}\)	dataAll your rows, each with a score from a judge, a checker or a person.
\(\mathbb{E}_{(x,y)\sim\mathcal{D}}\)“average over rows x, y from D”	mathThe average over the rows of the dataset; in practice, over the rows of a batch.
\(x\), \(y\)	dataA row's prompt and answer.
\(q(x,y)\)“q of x, y”, the quality tag	dataThe tag computed from this row's score (for example: judge 7 or more → `high`, else `low`) and written as text into its prompt. It sits on the right of the bar, so the model reads it and is never graded on it.
\(\log p_\theta\big(y \mid q, x\big)\)	trainedThe model's log-probability of the answer, having read the tag and the prompt: the #1 loss, with one extra piece of prompt.
\(-\)	mathMakes the loss positive, as in #1.
The rule at use time
\(y \sim p_\theta(\cdot \mid \dots)\)“y drawn from the model”	generatedThe model writes the answer. The dot means “any answer”: the model's whole spread of possible answers.
\(q=\texttt{high}\)	settingAt use time the system always writes the tag “high”, which selects the good half of what the model learned.

\(p_{\text{data}}(y\mid x)\)	dataHow often answer \(y\) appears for prompt \(x\) in the data, tags ignored (A 70%, B 30%).
\(p(\texttt{high}\mid x, y)\)	dataThe chance that this answer was tagged “high”: 1 for a good answer with perfect tags, 0 for a bad one.
\(p(\texttt{high}\mid x)\)	dataThe overall share of rows tagged “high” for this prompt (0.70 here). Dividing by it rescales the result so it adds up to 1.
\(p(y\mid x, \texttt{high})\)	resultWhat a well-trained model writes when asked for “high”: the data's answers, reweighted toward the ones that were tagged high.
Bayes' rule	mathThe general law \(p(A\mid B) = p(B\mid A)\,p(A)/p(B)\): it turns “how likely is the tag, given the answer” into “how likely is the answer, given the tag”.

\(p\)	dataHere a plain number, not a model: the share of good rows in the data (0.6).
\(n\)	dataThe tag noise: the share of rows whose tag is wrong (0.1 = one in ten).
\(p\,(1-n)\)	mathGood rows that were correctly tagged high.
\((1-p)\,n\)	mathBad rows that were wrongly tagged high.
\(P(\text{good}\mid\texttt{high})\)	resultAmong the rows tagged high, the share that is really good: what “high” will produce.

The idea in plain words

See it work

Next-token trainer

Where the training signal comes from

A small example

The conversation

What the model actually reads

What a training batch looks like

The loss

One training step on the GPU

1The batch: rows become one grid

2Forward pass: every position at once

3Loss: from each row to one number

4Backward pass: from one number back to every weight

5Update: every weight moves once

The maths behind it

The probability of a whole answer is a product

Maximum likelihood: choose the weights that make the data most probable

Find the best knob

Logs turn the product into a sum

Flip the sign and average: that is the loss

What one training step does: the gradient

SFT copies proportions, and that is W1

Where is the lowest loss?

Watch out for

It copies bad habits W1

Mistakes snowball W2

The ceiling is the teacher W3

Silent truncation

Forgetting to mask the prompt

Memorising a small dataset

The idea in plain words

See it work

Filter ten sampled answers

Where the training signal comes from

A small example

Kept: the checker passes it, the judge gives 9

Rejected: wrong total, the judge gives 2

The judge's own exchange

What a training batch looks like

The loss

One training step on the GPU

1The batch

2Forward pass

3Loss: from each row to one number

4Backward pass: the slope at every answer token

5Update

The maths behind it

Filtering edits the distribution the model is forced to copy

A real judge makes mistakes: how much bad slips through?

Judge quality calculator

Rejection sampling: training on your own successes

Why it cannot beat the grader

Watch out for

The grader becomes the ceiling

Fewer rows, more memorising

Judge steps, not whole runs

It teaches the absence of bad

Easy prompts take over

Checker loopholes

The idea in plain words

See it work

Hard target vs soft target

Where the training signal comes from

A small example

The teacher answers

The student, before and after

What the logit view adds, for the first token

What a training batch looks like

The loss

Sequence-level \(\mathcal{L}_{\text{seq}}\)

Logit-level \(\mathcal{L}_{\text{KD}}\)

One training step on the GPU

1The batch

2Forward pass

3Loss: from each row to one number

4Backward pass: the slope at every answer token

5Update

The maths behind it

Temperature softens a distribution

Symbol	What it means
The loss of one stage
\(k\)“k”, the stage number	mathWhich stage of the curriculum: 1, 2, 3. Written as a superscript in brackets, \((k)\), it is a label (“the stage-\(k\) version”), not a power.
\(\mathcal{L}^{(k)}(\theta)\)“L stage k of theta”	resultThe loss during stage \(k\): ordinary SFT, nothing new.
\(\theta\)	trainedThe model's weights. Each stage starts from where the previous stage left them.
\(\mathbb{E}_{(x,y)\sim\mathcal{D}^{(k)}}\big[\dots\big]\)	mathThe average of the bracket over rows drawn from the stage-\(k\) mix.
\(-\log p_\theta(y\mid x)\)	trainedThe #1 loss on one row.
\(\mathcal{D}^{(k)}\)“D stage k”	dataThe mix that stage \(k\) draws its rows from.
\(\rho\)“rho”, the replay share	settingThe fraction of stage-\(k\) rows drawn from easier bands, for example 0.25. \(\rho = 0\) gives pure bands.
\((1-\rho)\,\mathcal{B}_k + \rho\,\mathcal{B}_{<k}\)	mathA recipe, not an addition of numbers: take a share \(1-\rho\) of the rows from band \(k\) and a share \(\rho\) from the easier bands.
The difficulty bands
\(\mathcal{B}_k\)“B sub k”, band k	dataThe rows whose difficulty falls in the \(k\)-th range, for example levels 1–2, then 3, then 4–5.
\(\mathcal{B}_{<k}\)“B before k”	dataEvery easier band: all bands before \(k\).
\(\mathcal{D}\)	dataThe whole dataset, all levels together.
\(\{(x,y)\in\mathcal{D} : \dots\}\)“the set of rows in D such that …”	mathSet notation: curly brackets collect every row that satisfies the condition after the colon.
\(d(x,y)\)“d of x, y”, difficulty	dataThe difficulty score of a row, from people, heuristics, a judge or a model's loss (step 4 below).
The update steps
\(i\)	mathThe step counter: \(i = 1\) is the first update of training, \(i = 2\) the second, and so on.
\(\theta_i\), \(\theta_{i+1}\)	trainedThe weights before and after step \(i\).
\(b_i\)	dataThe \(i\)-th batch of rows. Curriculum learning is a choice of which rows go into which \(b_i\).
\(\ell(\theta_i;\ b_i)\)“small l”	trainedThe SFT loss on that one batch, measured at the current weights. The semicolon reads “on”.
\(\nabla_\theta\)“nabla”, the gradient	mathThe gradient: for every weight, the slope of the loss. It points in the direction that makes the loss grow fastest.
\(\eta_i\)“eta sub i”, the learning rate	settingThe size of step \(i\). It usually shrinks toward the end of training (a schedule you choose).
\(-\,\eta_i\,\nabla_\theta\ell\)	mathMove a small step against the gradient, so the loss goes down. This is gradient descent, the engine under every technique on this page.

\(d(x,y)\)	resultThe difficulty of the row: higher means harder.
\(p_{\text{ref}}\)“p ref”, the reference model	frozenA fixed model used only to measure difficulty, often the model before fine-tuning. Its weights do not change.
\(T\)	dataHere: the number of tokens in the answer \(y\) (not a temperature or a number of steps).
\(y_t\), \(y_{<t}\)	dataThe answer token at position \(t\), and the answer tokens before it.
\(-\frac{1}{T}\sum_{t=1}^{T}\log\)	mathThe average surprise per token, exactly the #1 loss: a row the reference model finds surprising is a hard row.