Online reinforcement learning

Every technique in the three workbooks makes some answers more likely and others less. They differ in two things only: who wrote the answers the loss reads, and the weight each answer gets. Click a rung of the ladder.

“Action” is used at three sizes. For the loss, the smallest action is one token. For an agent, the natural unit is a turn, many tokens long. The reward usually judges the whole episode. Which of the three gets the credit is question Q4, and #20 is about it.

RL can only strengthen an action the model already takes now and then. If the right move has probability 0, no try ever contains it and no reward ever pays for it directly; at most it rises as a side effect of learning something related. That is why RL starts from a model that already roughly knows the task (usually an SFT model, sometimes a strong base model), and why tries are sampled with some randomness instead of always taking the favourite.

Steps 5 to 7 are SFT's training step with a weight in front of each try. Steps 2 to 4 are new: in SFT the text came from a file, while here it comes from the model being trained, and step 8 says it must be made again: after every update in the plain recipe, after a few when the batch is reused, as in PPO (#18).

1. The goal: the average reward

   What we want to raise is the reward the model earns on average, over everything it might write:

   $$J(\theta) = \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\big[r(x,y)\big] = \sum_{y}\pi_\theta(y\mid x)\,r(x,y)$$

   \(J(\theta)\)“J of theta”	resultThe objective: the expected reward of the model with weights \(\theta\). RL raises it; there is no right answer to compare with.
   \(\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\big[\;\cdot\;\big]\)	mathThe average of the bracket over the answers the model would write for \(x\), each counted as often as the model writes it.
   \(\sum_y\)	mathA sum over every possible answer. For a real model there are far too many to list, which is why the next steps replace it with a few samples.
   \(\pi_\theta(y\mid x)\)	trainedThe probability the model gives to answer \(y\). It is the only part the weights can change.
   \(r(x,y)\)	frozenThe reward of answer \(y\): a checker, a judge, a reward model. A fixed function of the text.

   three moves, rewards 0 / 1 / 0, probabilities 0.50 / 0.30 / 0.20 J = 0.50 × 0 + 0.30 × 1 + 0.20 × 0 = 0.30 the model reaches the flag 30% of the time

2. The reward has no slope; the probabilities do

   The answer is discrete text, picked by sampling. Nudging a weight does not change a written answer a little, so no slope reaches the weights through the reward, whether the reward is a program or even a neural reward model. What the weights do move smoothly is the probability of each answer, and those probabilities have slopes:

   $$\nabla_\theta J = \sum_{y} r(x,y)\;\nabla_\theta\,\pi_\theta(y\mid x)$$

   \(\nabla_\theta\)“nabla theta”, the gradient	mathFor every weight: how much the quantity after it changes when that weight is nudged up.
   \(\nabla_\theta\,\pi_\theta(y\mid x)\)	trainedHow each weight changes the probability of answer \(y\). Backpropagation computes it.
   \(r(x,y)\)	frozenA plain number in front: rewarded answers count, unrewarded ones do not.

   The formula is exact but useless as it stands: it needs every possible answer.

3. The log trick: from every answer to a few tries

   A one-line identity, the slope of a log, rewrites the slope of a probability as the probability times the slope of its log: \(\nabla\pi = \pi\,\nabla\log\pi\). Put it in, and the sum over answers weighted by their probability becomes an average over answers the model samples:

   $$\nabla_\theta J = \sum_{y}\pi_\theta(y\mid x)\,r(x,y)\,\nabla_\theta\log\pi_\theta(y\mid x) = \mathbb{E}_{y\sim\pi_\theta}\big[\,r(x,y)\,\nabla_\theta\log\pi_\theta(y\mid x)\,\big] \approx \frac{1}{K}\sum_{i=1}^{K} r_i\,\nabla_\theta\log\pi_\theta(y_i\mid x)$$

   \(\nabla_\theta\log\pi_\theta(y\mid x)\)	trainedSFT's direction for answer \(y\): the change to the weights that makes it more likely. The SFT workbook computes exactly this.
   \(\approx\)	math“Is estimated by”. The average over K samples is right on average and noisy for any single group.
   \(y_i,\ r_i\)	generatedThe i-th try and its reward.

   Read it as a recipe. Sample K tries. For each one, take SFT's direction for that try, multiplied by its reward, and average. On average, a small step in that direction raises the average reward; any single group can point the wrong way. This is the policy gradient theorem in its simplest, one-step form, and the algorithm that follows it is REINFORCE (Williams, 1992).

4. The baseline is free

   Subtract the same number \(b\) from every reward and, on average, the direction does not change:

   $$\mathbb{E}_{y\sim\pi_\theta}\big[\,b\,\nabla_\theta\log\pi_\theta(y\mid x)\big] = b\sum_y \nabla_\theta\,\pi_\theta(y\mid x) = b\,\nabla_\theta\!\sum_y\pi_\theta(y\mid x) = b\,\nabla_\theta 1 = 0$$

   A push of the same size \(b\) on every answer, each counted as often as the model writes it, is a push on their total, and the total is always 1: it adds nothing. What the baseline changes is the noise. With rewards of 0 or 1 and no baseline, the failed tries get no push at all, and the successful ones are pushed up even when everyone succeeds. With \(b\) set to what the model usually gets, “better than usual” goes up and “worse than usual” goes down. The one rule: \(b\) may depend on the task and on the other tries, never on the try it is subtracted from, or the average changes. That is why #16 leaves each try out of its own baseline; the plain group mean, which includes the try, shrinks the average step by \(\frac{K-1}{K}\) without changing its direction.

   rewards 0 / 1 / 0 / 0, the model's true average reward J = 0.30 no baseline weights 0.00 1.00 0.00 0.00 the three failures get no push baseline b = J = 0.30 weights −0.30 +0.70 −0.30 −0.30 failures down, the success up, same average direction

   A baseline computed from the group itself (#16, #19) has one more property: a group in which every try got the same reward gets no push at all. A fixed baseline or a critic (#18) would still push such a group.

5. A loss whose slope is that recipe

   Trainers minimise a loss, so write one whose gradient is minus the recipe. Treat each advantage as a plain number (no slope flows into the checker) and use SFT's log-probability:

   $$\mathcal{L}(\theta) = -\frac{1}{K}\sum_{i=1}^{K} A_i\,\log\pi_\theta(y_i\mid x) \qquad\Longrightarrow\qquad -\nabla_\theta\mathcal{L} = \frac{1}{K}\sum_{i=1}^{K} A_i\,\nabla_\theta\log\pi_\theta(y_i\mid x)$$

   This is called a surrogate loss: its value does not track progress (it can be negative, with a group baseline it is exactly 0 for any group whose tries all got the same reward, successes or failures, and a new sample of tries changes it completely), but its slope is the one that raises the reward.

6. At the logits: the advantage times p − y

   The backpropagation page shows that the slope of −log p(target) at the logits is \(p - y\), the error slope. In the RL loss each try's −log p carries the weight \(A_i/K\), so:

   $$g_z = \frac{A_i}{K}\,\big(p - y\big)$$

   That is the whole difference from SFT at the point where the backward pass starts. A positive advantage gives SFT's push toward the token the try wrote. A negative advantage gives the same push reversed, away from it. From the logits down, every box, every slope and the update are exactly the ones on the backpropagation page.

• Q2 Noise: a few tries, one number each

  Four tries are a tiny sample, so two groups on the same task can point in very different directions. A baseline removes part of the noise: the other tries' average (#16), a critic that predicts the reward still to come (#18), or the group's mean, with the advantages then divided by its spread (#19).

• Q3 Waste: each batch of tries is used once

  Generating tries is the slow part, and the plain recipe throws them away after a single small step. PPO (#18) reuses one batch for several steps, and a clip stops pushing a token once its probability ratio to when the batch was written has left 1 ± ε in the direction its advantage wants, so the steps stay where the old tries still say something reliable. GRPO (#19) keeps the clip.

• Q1 Q3 Drift and hacking: the model finds whatever pays

  A reward that pays for something easier than the task gets the easier thing. The first defence is an honest checker (#17). The second is a KL leash, a cost for moving far from the starting model (#18, #19), so the model improves without turning strange.

• Q4 Credit: one reward, thirty turns

  A session that reads the secret gets one reward, and every one of its turns shares it, the wasted ones too. Multi-turn agent RL (#20) compares turns that faced the same situation, and adds progress rewards built so that they cannot be farmed.

1. The plain recipe, with a baseline that leaves each try out of its own average#16 REINFORCE, RLOO
2. A reward a program computes from the lab's ground truth, built to be hard to fool#17 Verifiable rewards
3. Bigger, safer steps: a clip, a learned critic, a KL leash#18 PPO
4. PPO without the critic: the group of tries is the baseline. The practical choice when a checker exists#19 GRPO and its fixes
5. Whole sessions as tries: tool outputs masked, credit cut per turn#20 Multi-turn agent RL

Three decisions: which tasks the model practises on, who scores the tries, and how many tries each task gets.

1. The policy gradient, from section 3

   $$\nabla_\theta J = \mathbb{E}_{y\sim\pi_\theta}\big[\,(r(x,y) - b)\,\nabla_\theta\log\pi_\theta(y\mid x)\,\big]$$

   Raising the average reward means following, on average over the model's own answers, SFT's direction weighted by reward minus baseline. Any baseline that does not depend on the answer it is subtracted from leaves the average unchanged (section 3, step 4).

2. Why leave the try out

   With the plain group mean, \(b = \frac{1}{K}\sum_j r_j\) contains \(r_i\) itself, so the baseline depends on the try: a lucky try raises its own baseline. Leaving it out removes that dependence, and the estimate is exactly unbiased. The two differ only by a constant factor:

   $$r_i - \frac{1}{K}\sum_{j} r_j \;=\; \frac{K-1}{K}\Big(r_i - \frac{1}{K-1}\sum_{j\neq i} r_j\Big)$$

   \(\frac{1}{K}\sum_j r_j\)	resultThe group's mean, try \(i\) included.
   \(\frac{1}{K-1}\sum_{j\neq i} r_j\)	resultThe leave-one-out baseline: the other tries' mean.
   \(\frac{K-1}{K}\)	mathA constant: 0.75 for K = 4. It only rescales the step, like a smaller learning rate.

   So the group-mean version is RLOO with a learning rate \(\frac{K-1}{K}\) times smaller. That is why the two are often used interchangeably (with Adam, a constant factor on the gradient is almost entirely cancelled), and why GRPO (#19) can use the group mean.

3. How noisy is one group?

   For a task where the model succeeds with probability \(s\), a group of K tries holds at least one success and at least one failure with probability

   $$P(\text{signal}) = 1 - s^{K} - (1-s)^{K}$$

   \(s\)	resultThe model's current success rate on the task.
   \(s^K\)	mathThe chance that all K tries succeed: every advantage 0.
   \((1-s)^K\)	mathThe chance that all K tries fail: every advantage 0.

   success rate s K = 2 K = 4 K = 8 K = 16 0.05 0.10 0.19 0.34 0.56 0.30 0.42 0.75 0.94 1.00 0.95 0.10 0.19 0.34 0.56

   Tasks the model almost always fails, or almost always solves, teach much less often, unless K is large. RL learns most from tasks at the edge of what the model can do.

4. What one step does at the logits

   At each graded position of try \(i\) the slope on the logits is \(\frac{A_i}{K}(p - y)\), the error slope of the backpropagation page times one number. Training moves the logits against it. A positive advantage raises the written token's logit and lowers every other logit in proportion to its probability, exactly as SFT does. A negative advantage does the reverse: the written token's logit falls and the others rise. This is the push SFT never makes, the same one DPO makes on a rejected answer.

   the login lab, logits of A / B / C, averaged over the four tries (RLOO) slope g +0.167 −0.250 +0.083 step −1 × g −0.167 +0.250 −0.083 B's logit rises, A's falls most: it failed twice

• Groups that teach nothing

  All tries succeed or all fail: every advantage is 0, and the generation was wasted. Track the share of such groups. Replace solved tasks with harder ones, and drop tasks the model never solves until it is stronger.

• Exploration dies

  As the model grows confident, its K tries become identical and the groups stop carrying signal. Sample at a temperature near 1, and watch the variety of the tries.

• Noisy steps

  A few tries per task give a rough direction. Use many tasks per batch and a small learning rate, and judge progress by the average reward over many steps, not by any single one.

• The loss is not progress

  The surrogate loss jumps with every new sample of tries. Plot the average reward, on training labs and on held-out ones.

• Nothing holds the model back

  Plain REINFORCE has no leash: a long run can drift far from the starting model, in style and in skills the reward never checks. The published RLOO and TRL's RLOOTrainer subtract a KL penalty from the reward; PPO adds one, and GRPO can (#18, #19).

• The checker is the target

  Whatever the reward pays for is what the model learns, gaps included. Before scaling up, read a sample of the tries that score highest. #17 is about building a reward that pays for the task and nothing easier.

Three decisions: what proves success, how the score is built, and what the checker reads.

1. The reward needs no slope

   The policy gradient \(\mathbb{E}\big[(r - b)\nabla_\theta\log\pi_\theta(y\mid x)\big]\) uses the reward's value only. It never asks how the reward would change if the try changed a little. So the reward can be any function at all, however discontinuous: a string match, a test suite, “did the listener get a call?”. This is what makes RL usable on tasks a gradient could never see through.

2. The model maximises the reward, not your intent

   $$\pi^{\star} = \arg\max_{\pi}\;\mathbb{E}_{y\sim\pi}\big[r(x,y)\big]$$

   \(\arg\max_\pi\)	math“The \(\pi\) that makes what follows largest.”
   \(\pi^\star\)	resultThe best policy for this reward: where training is heading.

   If some behaviour \(y_{\text{cheap}}\) earns more than the honest one, \(r(y_{\text{cheap}}) > \mathbb{E}[r\mid\text{honest}]\), then wherever the policy samples it at all, the gradient raises it (its expected push is \(\pi(y_{\text{cheap}})\,(r(y_{\text{cheap}}) - \mathbb{E}_\pi[r])\)), \(\pi^\star\) ends on it, and nothing in the maths knows that it is cheating. Goodhart's law in one line. The only fix is in \(r\).

3. Why F1 stops the spray

   Prove the three real findings and add \(n - 3\) unproven ones to the report. Recall is 1; precision is \(3/n\). A reward that counts mentions gives 1 whatever \(n\) is. F1 gives

   $$F_1 = \frac{2\,|V^{+}\cap E|}{|V|+|E|} = \frac{2\times 3}{n + 3}$$

   findings reported n 3 6 12 30 counts mentions 1.00 1.00 1.00 1.00 F1 1.00 0.67 0.40 0.18

4. Binary or partial credit: how often a group carries signal

   A lab hides 3 findings, and each try finds each one with chance \(q\). With pass or fail (here: all three findings proven, or nothing), a try succeeds with \(s = q^3\), and a group of K carries signal with probability \(1 - s^K - (1-s)^K\). With a share per finding, a group is silent only when every try found the same number of findings:

   $$P(\text{signal}) = 1 - \sum_{c=0}^{3} P(c)^K \qquad P(c) = \binom{3}{c}\,q^c\,(1-q)^{3-c}$$

   \(c\)	mathHow many of the 3 findings a try found.
   \(P(c)\)	resultThe chance a try finds exactly \(c\) of them.
   \(\binom{3}{c}\)	math“3 choose c”: the number of ways to pick which \(c\) findings.

   q = 0.2, K = 4 pass or fail s = 0.2³ = 0.008 P(signal) = 1 − 0.008⁴ − 0.992⁴ = 0.03 a share each P(c) = 0.512, 0.384, 0.096, 0.008 P(signal) = 0.91

   On a hard lab pass-or-fail almost never speaks, and partial credit almost always does. That is the case for partial credit, provided false positives cost something.

5. A proof that cannot be guessed

   A secret of \(m\) random bits is guessed with probability \(2^{-m}\): about \(3\times10^{-39}\) for 128 bits. A fixed flag, the same in every run, stops proving anything once the model has seen it: after a few successes it can be recited without any exploit. Generate the secret per run.

• The answer key inside the sandbox

  If the ground-truth file, the checker's code or the secret can be read from where the model's tools run, the model will learn to read it. Keep them on the checker's side of the wall.

• Trusting the model's own words

  An agent's own analysis may say exploit_succeeded: true. As a reward it teaches the model to say so. Pay only for evidence the environment produced.

• A fixed or guessable secret

  The same flag in every run, or a predictable format, is memorised after a few successes. Generate it per run.

• Labs that leak between tries

  One try's exploit leaves files, sessions or log entries that the next try finds. One fresh lab per try.

• Flaky labs

  Timeouts, rate limits and slow start-ups turn into random rewards. Fix the lab before training on it. A timeout the try caused (a loop, a stall) counts as a failure, never as a missing value, or stalling gets rewarded; one the lab caused is retried or masked.

• Labs, never real targets

  Every try is a real attack. Train only against labs built for it, in an isolated network, and never put real hosts, scan output or credentials into the task set. Scope costs teach the rules; the hard guardrails stay in code.

Three decisions: what scores the tries, what the critic is, and where the KL leash goes.

1. Reusing old tries: importance sampling

   An average over what the current model would write can be computed from tries an older model wrote, by weighting each try with the ratio of the two probabilities:

   $$\mathbb{E}_{y\sim\pi_\theta}\big[A(y)\big] = \sum_y \pi_\theta(y)\,A(y) = \sum_y \pi_{\text{old}}(y)\,\frac{\pi_\theta(y)}{\pi_{\text{old}}(y)}\,A(y) = \mathbb{E}_{y\sim\pi_{\text{old}}}\big[\rho(y)\,A(y)\big]$$

   Its slope at the first pass, where \(\pi_\theta = \pi_{\text{old}}\), is \(\nabla\rho = \rho\,\nabla\log\pi_\theta = \nabla\log\pi_\theta\): exactly the policy gradient of #16. For a whole try's ratio the identity holds at any distance, but the ratios grow uneven and a few tries with large ratios dominate the estimate. PPO uses each token's own ratio instead of the product over the try, which keeps the weights small; the price is that the per-token objective matches the true one only near \(\pi_{\text{old}}\), because it ignores that the new policy would also reach different prefixes. That is the real reason to stay close.

2. A trust region, cheaply

   The fix is to stay close to \(\pi_{\text{old}}\), where the old tries are still a fair sample. TRPO (Schulman et al., 2015) imposes it as a hard constraint on the KL divergence between the two policies, which needs second-order maths. PPO (Schulman et al., 2017) gets most of the effect with the clip and plain gradient descent.

3. The min, case by case

   $$\min\big(\rho A,\;\mathrm{clip}(\rho,1-\varepsilon,1+\varepsilon)A\big) = \begin{cases} \min(\rho,\,1+\varepsilon)\,A & A > 0 \\ \max(\rho,\,1-\varepsilon)\,A & A \lt 0 \end{cases}$$

   For a good token the objective rises with ρ until \(1+\varepsilon\), then goes flat: no slope, no push. For a bad token it rises as ρ falls, until \(1-\varepsilon\), then goes flat. On the other side of 1 nothing is clipped, so a move in the wrong direction is always pushed back. Part 1 of the lab draws both curves.

4. GAE: from a critic's predictions to per-token credit

   $$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t) \qquad \hat A_t = \sum_{l\ge 0}(\gamma\lambda)^l\,\delta_{t+l}$$

   With λ = 0 the advantage is the one-token surprise \(\delta_t\): little noise, but only as right as the critic. With λ = 1 (and γ = 1) the sum telescopes to \(\sum_{l} r_{t+l} - V(s_t)\): the actual return minus the critic's baseline, with no bias and all the noise of a whole try. λ ≈ 0.95 sits between the two (Schulman et al., 2016).

   station 5's try, first token λ = 1 Â₁ = 0.04 + 0.18 + 0.25 + 0.19 = 0.66 = (sum of rewards 0.96) − V₁ 0.30 λ = 0.95 Â₁ = 0.60 λ = 0 Â₁ = δ₁ = 0.04 t1 barely changed the critic's prediction, so the credit goes to later tokens

5. What the KL leash allows

   With the KL penalty, PPO maximises the reward minus β times the drift from the reference. That goal has a closed-form best policy, derived in the DPO maths:

   $$\pi^{\star}(y\mid x) = \frac{1}{Z(x)}\,\pi_{\text{ref}}(y\mid x)\,\exp\!\Big(\frac{r(x,y)}{\beta}\Big)$$

   three moves, π_ref = 0.50 / 0.30 / 0.20, only the second rewarded (r = 1) β = 1 π_ref × e^(r/β) = 0.50 / 0.82 / 0.20 normalised: π* = 0.33 / 0.54 / 0.13 β = 0.2 π_ref × e^(r/β) = 0.50 / 44.5 / 0.20 normalised: π* = 0.011 / 0.985 / 0.004 β → 0 π* puts everything on the rewarded move

   A large β keeps the model near its starting habits; a small one lets the reward take over. With a checker that is hard to fool, a small β is much less risky, though drift and lost variety remain. With a reward model that can, β is what stands between the model and the reward model's mistakes.

• A critic that knows nothing yet

  A fresh critic gives arbitrary baselines, so the first advantages are noise. Initialise it from the reward model or the SFT model, and consider training it alone for a few batches first.

• Too many passes

  Each pass over the same batch pushes it further from the policy that wrote it. Watch the clip fraction and the KL to \(\pi_{\text{old}}\); if they climb, use fewer passes or a smaller learning rate.

• Reward model over-optimisation

  With a learned reward, its score keeps rising while real quality peaks and then falls (Gao et al., 2023). Check a held-out judge or people, and keep β high enough. With a checker this risk is the checker's gaps instead (#17).

• Many moving parts

  Four models, two losses, GAE, whitening, two clips: PPO for language models has dozens of details that matter (Huang et al., 2024). Start from a tested implementation, not from scratch.

• Tries cut off at the length limit

  A try that hits the length limit gets scored as if it had finished. Penalise or drop tries without an end token, or the model learns that rambling until the cut is safe.

• Memory

  The critic is often about as large as the policy. If it does not fit, GRPO (#19) removes it, at the cost of several tries per task.

Four decisions: the reward, how advantages are scaled, which groups train, and what holds the model back.

1. The group mean is a critic made of samples

   PPO's critic learns \(V(x) \approx \mathbb{E}_{y\sim\pi_\theta}[r(x,y)]\). GRPO estimates the same number by sampling, \(V(x) \approx \frac{1}{G}\sum_j r_j\), at the cost of G tries per task and with no network to train. It gives one baseline per task, not one per token: every token of a try shares the try's advantage, as in #16. For short answers that loses little; for long agent sessions it is the credit problem of #20.

2. What dividing by the spread does

   Dividing by the std rescales each group so its advantages have a spread of 1. Every task with any signal then moves the model by a similar amount, whatever its difficulty. But groups with a small spread, nearly always solved or nearly always failed, are amplified most: the lone success in 16 gets +3.75. Liu et al. (2025, “Dr. GRPO”) show this, together with the length normalisation below, biases the update, and that removing both gives a cleaner policy gradient with no loss of performance.

3. The length normalisation and its bias

   $$\text{per-token weight under GRPO} = \frac{\hat A_i}{G\,|o_i|} \qquad \text{under DAPO} = \frac{\hat A_i}{\sum_{j=1}^{G}|o_j|} \qquad \text{under Dr. GRPO} = \frac{\hat A_i}{G\,L_{\max}}$$

   The same \(\hat A_i\) is used in all three to compare the denominators; Dr. GRPO's own advantage is \(r_i - \text{mean}\), with no division by the spread.

   \(\sum_j |o_j|\)	mathAll the tokens of the group, as in DAPO's formula; implementations divide by the tokens of the whole batch.
   \(L_{\max}\)	settingA constant, such as the maximum length of a try.

   GRPO gives every try the same total push whatever its length, so for a negative advantage each extra token dilutes the penalty. DAPO and Dr. GRPO give every token the same weight. The worked example in station 6 puts numbers on it.

4. The k3 estimate of the KL

   With \(u = \pi_{\text{ref}}(o_t)/\pi_\theta(o_t)\) for a token the model sampled, \(\hat D = u - \log u - 1\). Since \(u - \log u - 1 \ge 0\) for every \(u > 0\), it is never negative, unlike the plain \(-\log u\). And since \(\mathbb{E}_{o\sim\pi_\theta}[u] = 1\), its average is exactly the KL divergence of \(\pi_\theta\) from \(\pi_{\text{ref}}\) (Schulman, 2020). One subtlety: used as a loss term, with the gradient flowing only through \(\pi_\theta\) in \(u\), its expected gradient is that of the KL in the other direction, \(\mathrm{KL}(\pi_{\text{ref}}\,\|\,\pi_\theta)\). Both are 0 at the start and pull toward the reference; multiplying the term by the ratio ρ, as recent TRL does by default (use_bias_correction_kl, following DeepSeek-V3.2), gives the gradient of \(\mathrm{KL}(\pi_\theta\,\|\,\pi_{\text{ref}})\).

   u = π_ref / π_θ 0.5 0.8 1.0 1.25 2.0 D̂ = u − ln u − 1 0.19 0.02 0.00 0.03 0.31 slope pulls the token down down none up up

5. DAPO's four fixes, and GSPO

   DAPO (Yu et al., 2025) changes four things and drops the KL term. Clip-higher: \(\varepsilon_{\text{high}} = 0.28 > \varepsilon_{\text{low}} = 0.2\), so a rare token that paid off can grow faster, which delays entropy collapse. Dynamic sampling: keep only groups with \(0 \lt \#\text{correct} \lt G\). Token-level loss: the division by all tokens above. Overlong shaping: tries cut at the length limit are masked, and a soft penalty grows as a try approaches the limit.

   GSPO (Zheng et al., 2025) moves the ratio from tokens to tries: \(s_i = \big(\pi_\theta(o_i)/\pi_{\text{old}}(o_i)\big)^{1/|o_i|}\), the geometric mean of the token ratios, clipped per try. Every token of a try now shares one weight, built from the average of its tokens' log-ratios, so single-token noise no longer accumulates over a long try, and the clip keeps or drops whole tries. This steadies training on long outputs and on mixture-of-experts models.

• Entropy collapse

  As tries become identical, groups stop disagreeing and learning stalls on a single habit. Watch the entropy and the share of mixed groups; use clip-higher and a temperature near 1.

• Silent groups

  All-correct and all-wrong groups waste their generation. A pass-rate curriculum, which keeps labs at the edge of the model's skill, avoids drawing them; dynamic sampling keeps them out of the batch, at the cost of more generation.

• Inflated rare wins

  Dividing by a tiny spread turns one lucky success (or one unlucky failure on an easy task) into a huge advantage. If the reward can be lucky (a flaky lab), drop the division or put a floor under the spread.

• Length drift

  Under per-try normalisation, wrong answers grow longer. Use a token-level loss, an overlong penalty, and watch the mean length of right and wrong tries separately.

• No KL, no anchor

  β = 0 is common with a trustworthy checker. With a learned reward or a judge, keep a KL term: nothing else stops the model drifting into their blind spots.

• Format rewards that win alone

  A small reward for a well-formed report is useful early. If it can be maxed out without doing the task, the model will settle for it: keep it small, and check that the checker's part keeps rising.

Four decisions: the environment, the outcome reward, how credit reaches the turns, and the budget.

1. Return: the reward from here on

   $$G_k = \sum_{j\ge k}\gamma^{\,j-k}\,r_j$$

   \(G_k\)	resultThe return from turn \(k\): every reward still to come. (Not the group size \(G\).)
   \(\gamma\)	settingThe discount, between 0 and 1: how much a reward later counts less than one now. Often 1 for the session's outcome; GiGPO uses 0.95 for its turn-level returns, so of two visits to the same state, the one closer to success scores higher.

   With one reward at the end and γ = 1, every turn's return is the session's reward. That is session-level credit, and the policy gradient with it is unbiased: on average over many sessions, good turns are pushed up more than bad ones.

2. Why one reward gets noisier with every turn

   A turn's advantage under session-level credit is \(R - b\), and \(R\) depends on every other turn's luck as well as on this turn. With 30 turns, one turn's push carries the randomness of 29 others. The average is right; any single session says little about any single turn. The noise grows with the length of the session, and so does the number of sessions needed.

3. Same-state groups are a baseline per state

   $$\hat A^{S}_{i,k} = G_{i,k} - \frac{1}{|\mathcal{S}(s_{i,k})|}\sum_{(j,l)\in\mathcal{S}(s_{i,k})} G_{j,l}$$

   \(\mathcal{S}(s)\)	resultEvery turn in the group, from any session and at any point in it, that started from state \(s\), the turn itself included.
   \(G_{j,l}\)	resultThe return of that turn.

   A leave-one-out version, the mean over the other turns from that state (counting only turns from other sessions: a revisit within the same session depends on the action), depends only on the state and not on the action taken there: it changes nothing on average and removes the luck of everything that happened before the state. The plain mean, as here and in GiGPO, includes the turn itself and shrinks each push by \((n-1)/n\) for a state reached \(n\) times: half for a pair, nothing at all for a state only one turn reached (as #16 showed for the group mean). It needs states that repeat, which is why GiGPO matches the environment's state rather than the whole history. It keeps the session-level part as a stable, whole-session signal; the turn-level part adds local detail where states repeat.

4. Shaping that cannot be farmed

   Add \(F_k = \gamma\Phi(s_{k+1}) - \Phi(s_k)\) to each turn's reward. Over a whole session the discounted additions telescope:

   $$\sum_{k=0}^{T-1} \gamma^{k} F_k = \gamma^{T}\Phi(s_T) - \Phi(s_0) \qquad (\gamma = 1:\ \ \Phi(s_{\text{end}}) - \Phi(s_{\text{start}}))$$

   With γ = 1, the total depends only on where the session started and ended, never on the path (with γ < 1, also on when it ended). With γ = 1, a loop back to an earlier state earns exactly 0, and so does repeating a scan that changes nothing (with γ < 1, slightly less than 0). A naive bonus, such as +0.1 every time a tool output is judged to contain new information, does not telescope, and a long enough loop of slightly different scans beats solving the lab. Ng, Harada and Russell (1999) showed that potential-based shaping leaves the best policy unchanged. Strictly, that needs Φ = 0 at the end of every session; keeping the progress at the end, as here, also makes the reward prefer failures that got further. That is then a different objective, a partial credit that could trade some success rate for milestones: keep Φ small next to the outcome, or set it to 0 at the end for strict invariance.

   One more caveat: shaping only changes which turns get the credit if each turn is scored by its own return from there on (same-state groups, or a critic as in PPO). Under session-level credit every turn shares one advantage, and shaping only adds \(\Phi(s_{\text{end}}) - \Phi(s_{\text{start}})\) to the session's score; with \(\Phi = 0\) at the end it changes nothing at all.

   a session that loops n scans and never exploits, against one that solves the lab in 3 turns (under the naive bonus the solve earns 1 + 0.1 × 2: two of its tool outputs hold new information) n = 5 n = 15 n = 30 solves the lab outcome only 0.0 0.0 0.0 1.0 naive +0.1 per new info 0.5 1.5 3.0 1.2 ← farming wins from n = 13 potential-based 0.2 0.2 0.2 2.0 ← loops earn nothing

5. The echo trap

   Multi-turn RL can collapse in a way single answers rarely do. The RAGEN study (Wang et al., 2025) describes an echo trap: the agent settles on one repeated pattern of reasoning, the rewards in a group stop varying, and training becomes unstable. Their remedies (StarPO-S): train only on the tasks whose rewards vary most, add a critic (PPO, #18, was steadier than GRPO on two of their four environments, GRPO on one), drop the KL term and raise the upper clip, and watch the reward spread, the entropy and gradient spikes, not only the reward.

• Grading the tool outputs

  If observation tokens are not masked, the model is trained to write tool outputs itself: it learns to make them up. Check the masks on a real session before the first run.

• Prompt injection that pays

  Tool outputs come from the target and can contain instructions. If following one ever earns reward, RL strengthens the habit. Keep the agent's boundary for untrusted output, and include labs whose planted instructions lead to a cost.

• No one at the approval gate

  In production a person approves dangerous tools; in training there is no one. Run rollouts only in isolated lab networks, scope any automatic approval to the lab, charge for forbidden techniques in the reward, and keep the production gate as it is.

• Sessions that never end

  Without a turn and token budget, a stuck agent burns the batch. Timeouts count as failures; long outputs are truncated before they reach the context.

• Progress the agent can claim

  A progress score computed from the agent's own words (“I confirmed the injection”) is farmed like any self-report. Compute Φ from what the environment returned.

• A training scaffold that differs from production

  Train with the same prompts, tools and parsers the agent runs with, or the learned habits do not transfer. Evaluate on labs the agent never trained on.

1. Learning how RL works, or a small model and a simple checker#16 REINFORCE, RLOO
2. Tasks with ground truth, such as labs: build and test the checker before anything else#17 Verifiable rewards
3. A learned reward model, a need for per-token credit, and memory for four models#18 PPO
4. A checker and several tries per task: the practical default#19 GRPO
5. An agent that works in turns with real tools#20 Multi-turn agent RL

1. SFT on good sessions, so the model already uses the tools and the report formatA SFT family
2. Optional: a preference stage on pairs of sessions, a success against a failure#15 Trajectory pairs
3. Build labs with ground truth and a planted secret per run; test the checker on honest, sprayed and dishonest reports#17 Verifiable rewards
4. GRPO on single decisions first: choose the next tool, write the payload, judge a tool's output#19 GRPO
5. Then whole sessions, in isolated labs, with GiGPO-style turn-level credit and a turn budget#20 Multi-turn
6. Evaluate on labs never trained on: success rate, cost per success, scope violations, and regressions on general skills

Starting with single decisions is cheaper by orders of magnitude, and it shows quickly whether the checker pays for the right things. Whole sessions come once it does.