Backpropagation, box by box

Three words first

Loss, slope and local slope With money instead of neurons

Before any network, here is what the three words mean and why a slope is something you can multiply.

1

The idea in plain words

A model has many knobs: its weights and biases. Training turns each knob a little so that the loss gets smaller, then does it again, millions of times.

To know which way to turn a knob, you need its slope: if I push this knob up a little, does the loss go up or down, and by how much? A slope of +6 says the loss rises 6 times as fast as the knob, so turn the knob down. A slope of −6 says turn it up. A slope of 0 says this knob does not matter right now.

Nobody can see the slope of a knob deep inside a network directly. But each small box can say its own local slope, and the chain rule turns those local slopes into the slope of every knob.

2

The chain rule is changing money

You change euros into dollars, then dollars into yen. Every exchange office is a box, and its rate is its local slope.

Forward 1 €→× 2→ 2 $→× 100→ 200 ¥

Backward 1 € is worth 200 ¥◄× 2◄ 1 $ is worth 100 ¥◄× 100◄ 1 ¥ is worth 1 ¥

Forward: add 1 euro and you get 2 × 100 = 200 more yen. Going through two offices, the rates multiply.

Backward: start at the end and ask "what is 1 of this worth, counted in yen?". One yen is worth 1, which is where every walk starts. Cross the ×100 office and a dollar is worth 100. Cross the ×2 office and a euro is worth 200. At each box you multiply by its rate. That is the whole chain rule.

Now swap the names

Yen becomes the loss. Euros become a weight. The exchange offices become the boxes of the network. The question is the same: if I turn this weight a little, how much does the loss move? Backpropagation answers it by walking from the loss back to the weight, right to left, multiplying by each box's local slope.

All of backpropagation

Two rules carry the slope everywhere Multiply along a road · add where roads meet

Every network, from the toy below to a large language model, moves the slope with only these two rules. Only the local slopes differ from box to box.

1

Rule 1 · along a road, multiply

Forward x = 1→× 3→ 3→+ 5→ 8→× 2→ loss = 16

Backward slope at x 6◄× 3◄ 2◄× 1◄ 2◄× 2◄ slope at the loss 1

Going back, each box multiplies the slope by its local slope. The + 5 box has local slope 1, because adding 5 does not change how fast anything moves. So the slope at x is 1 × 2 × 1 × 3 = 6.

check x = 1.01 → 3.03 → 8.03 → loss 16.06 the loss moved 0.06 = 6 × 0.01 ✓

2

Rule 2 · where roads meet, add

Here x is used twice. Nudging it moves the loss along both roads, so its slope is the sum of the two roads: 1 × 2 + 1 × 3 = 5.

check x = 1.01 → 2.02 + 3.03 = 5.05 the loss moved 0.05 = 5 × 0.01 ✓

In a network this happens everywhere: every neuron's output feeds every neuron of the next layer.

Both rules in one sentence

The slope of a knob = add up, over every road from the knob to the loss, the product of the local slopes along that road. Backpropagation never lists the roads: it walks right to left once, and every box reuses the slope that already reached it.

The lab · one SFT training step, box by box

A tiny language model, every box open 2 inputs · 2 hidden layers of 2 neurons with ReLU · 3 logits · plain SFT loss

The prompt is The capital of France is. Its last token reaches this network as two numbers. Two hidden layers of two neurons turn them into three logits, one per candidate next token. The SFT example says the right next token is Paris, but the model starts out betting on Rome.

Input: 2 numbers for the last prompt token
Hidden: 2 layers × 2 neurons: weights, bias, ReLU
Output: 3 logits: Paris, Rome, Lyon
Loss: Plain SFT: −ln p(right token)
Knobs: 21: 14 weights and 7 biases

1

Walk the slope back, box by box

The backward pass, right to left

Every neuron is opened into its boxes: multiply by weight, add plus bias, keep only positives. Press Next box ◄ to carry the slope from the loss back to the inputs, one box at a time. Click any box to read its rule with its own numbers, then nudge it and check the slope against the real change in the loss.

2

Five things to try

Step the whole road once. Every amber number is one multiplication or one addition of the numbers right next to it. Nothing else happens anywhere.
Open n4's gate. Its sum is negative, so the gate is closed: a slope of −0.710 arrives and 0 passes, so n4's weights and bias learn nothing from this example. Raise x₂ past 1.00 and the gate opens: step again and the slope now flows into n4.
Nudge a weight. Pick the weight n3→Paris, choose "its weight" and nudge by +0.05: the real change in the loss and slope × nudge agree. Nudge by +0.5 and they drift apart, because a slope only describes small moves.
Change the right token to Rome. The slopes at Paris and Rome swap sign: now Rome's score should rise and Paris's should fall. Every slope behind them changes to match, because the loss decides which way everything moves.
Press Train 1 step a few times. The loss falls and p(Paris) rises. Watch the magenta error slope p − y shrink toward 0: every other slope shrinks with it, because every other slope is built from it.

Inside an LLM · part by part

Every part of a language model, one lab each The level before, the part, the level after

The lab above is a plain network. A real LLM adds its own parts: embeddings, normalisation, attention, a wider MLP, a tied head, a loss mask, and the extras of post-training. Each part below gets its own lab, with the same boxes, the same buttons and the same colours.

Every lab shows three bands. On the left, the level before: where the part's inputs come from. In the middle, framed in teal, the part itself, opened into its small boxes. On the right, the level after: what reads the part's output. A short LM head and the loss close every lab, so a real slope reaches every box, and every nudge can be checked against the real loss.

The vectors are tiny, 2 or 3 numbers, and the output usually has two words (Paris, Rome), so every number fits on screen. The rules are the ones a full-size model uses. The parts come from two families: GPT-2 (position table, LayerNorm, GELU, tied head) and Llama (RoPE, RMSNorm, SwiGLU). No single model has all of them.

A · Input side
- 4.1 · Tokenizer → token ids Text to whole numbers
- 4.2 · Token embedding lookup Pick a row
- 4.3 · Position embedding Add where the token sits
B · One transformer block
- 4.4 · RMSNorm Rescale to a steady size
- 4.5 · Q, K and V projections Make query, key and value
- 4.6 · Dot product q · k How well they match
- 4.7 · Scale by 1/√d Keep the scores small
- 4.8 · Causal mask No peeking ahead
- 4.9 · Attention softmax Scores to weights
- 4.10 · Weighted sum of values Mix the values
- 4.11 · Output projection W_O Back to the stream
- 4.12 · Multi-head: split and concat Parallel heads
- 4.13 · Residual add The highway
- 4.14 · MLP up projection Widen the vector
- 4.15 · Activation: ReLU, GELU, SwiGLU A non-linear gate
- 4.16 · MLP down projection + residual Narrow back
C · Output side
- 4.17 · Final LayerNorm Normalise before the LM head
- 4.18 · LM head and weight tying Score every word
- 4.19 · Softmax + cross-entropy, box by box The SFT loss
- 4.20 · Loss mask over positions Grade only the answer
D · Post-training extras
- 4.21 · LoRA adapter A small trainable side road
- 4.22 · Frozen reference model (DPO) A copy that never learns

Inside an LLM · A · Input side

Tokenizer → token ids Text to whole numbers · the walk stops here

The tokenizer cuts the text into tokens and swaps each one for its number in the vocabulary. Those numbers are whole numbers: there is no “France plus a little”, so no slope can reach them. The backward pass ends one step later, at the row of the embedding table the id picked.

1

Walk the slope through it

The level before, the part, the level after

The id is the part (teal band). The lookup after it is the next level, then a short LM head and the loss. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Step to the end: the last box, the id, gets no slope chip. The slope stops at the two lookup boxes and goes into the table row E[France].
Switch the word to Italy: the same id box now picks another row, and only that row's numbers get a slope in the gradient table.
The dashed wire from the id to the lookups never lights up: an id chooses a row, it carries no slope.

Inside an LLM · A · Input side

Token embedding lookup Pick a row · only that row learns

The embedding table has one row of numbers per word. A lookup copies the row its id points to, so going back it copies the slope into that row and nowhere else. A word that appears twice gets the slope of both appearances, added.

1

Walk the slope through it

The level before, the part, the level after

Two tokens, each looked up in the same table, then averaged by the next level: a stand-in for attention. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

In the gradient table only E[France] and E[capital] get slopes. E[Italy] is not in the text, so it gets nothing this step.
Switch the second word to France: each number of E[France] is now read by two lookup boxes, one per position, and adds their slopes (rule 2, on a weight).
The next level averages the two positions, so each lookup receives half of the slope that reached the average.

Inside an LLM · A · Input side

Position embedding Add where the token sits · the position row learns too

A transformer reads every token at once, so it must be told where each one sits. The simplest way (GPT-2) keeps a second table with one row per position and adds that row to the token's row. An add box copies the slope to both inputs, so the token row and the position row get the same slope.

1

Walk the slope through it

The level before, the part, the level after

Two ids pick two rows, one from the token table and one from the position table, and an add box joins them. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Each token number and its position number get identical slopes: an add box copies.
Switch the position: the same token row is still picked (its slope shifts, because x changed), and a different position row learns.
Llama-style models use rotary embeddings (RoPE) instead: they rotate q and k, in every attention layer, by an angle that depends on the position, and the slope is rotated back by the same angle. RoPE has no table and no knobs: nothing about position is learned.

Inside an LLM · B · One transformer block

RMSNorm Rescale to a steady size · every input reaches every output

Before attention and before the MLP, each vector is rescaled to a standard size: divide it by its root mean square (rms), then multiply by a learned gain γ. Opened into small boxes: square, mean, square root, divide, multiply. Every divide box also reads the rms, so the slope reaches each input by two roads: directly, and through the size of the whole vector.

1

Walk the slope through it

The level before, the part, the level after

The level before is the token + position add; RMSNorm is in the teal band; a projection reads its output (the LM head stands in for it). Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Open a “÷ rms” box: its top gets the slope × 1/rms, its bottom gets the slope × (−x / rms²).
Open x₁: it feeds its own divide box and its square, and adds the two slopes.
Scaling the whole vector up does not change RMSNorm's output, so the slopes at x₁ and x₂ always point sideways: x₁ × (slope of x₁) + x₂ × (slope of x₂) comes out about 0.

Inside an LLM · B · One transformer block

Q, K and V projections Make query, key and value · three matrix boxes

From the same normalised vector, three weight matrices make three different vectors: a query (what this token is looking for), a key (what it offers to be found by) and a value (what it hands over). They are plain multiply-and-add boxes. What differs is where their slope comes from: q and k get theirs through the attention scores, v through the weighted sum.

1

Walk the slope through it

The level before, the part, the level after

One token computes its q, k and v (teal band); attention over this token and an earlier one is the next level. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Step to the three sums: q collects slope from both scores, k₂ only from score₂, v₂ only from the weighted sum.
n₁ feeds three multiply boxes, one per projection, and adds their slopes.
Each head here holds one number per vector (d = 1), so 1/√d = 1 and the scale box is left out. Part 4.7 shows it.

Inside an LLM · B · One transformer block

Dot product q · k How well they match · both sides are activations

The score of a token pair is q · k: multiply the query and the key number by number, then add. Unlike a weight box, both sides of each multiply are activations computed by the network, so the slope flows into both: q gets the slope × k, and k gets the slope × q.

1

Walk the slope through it

The level before, the part, the level after

Two keys (an earlier token and this one) meet this token's query; the scale, softmax and weighted sum are the next level. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Open q₁ × k₁,₁: each side gets the slope × the other side, query and key alike.
q₁ is used by two products, one per key, and adds their slopes. Each key number is used once.
There are no knobs inside: the weights that made q and k are further left (W_Q, W_K, part 4.5).

Inside an LLM · B · One transformer block

Scale by 1/√d Keep the scores small · a box with a constant

Dot products grow with the number of numbers in each vector: with 64 numbers per head, scores easily reach 10 or more. Softmax then puts nearly all the weight on one token, and its slopes nearly vanish. Multiplying every score by 1/√d keeps them where softmax still learns. Going back, the slope is multiplied by the same constant.

1

Walk the slope through it

The level before, the part, the level after

The same boxes as part 4.6, with larger numbers: scores about as big as a 64-number head's, so 1/√64 is the scale such a head would use (these vectors hold 2 numbers, so 1/√2 is offered too). The scale box is in the teal band. Switch the constant and compare the slopes that reach the scores. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

With “nothing”, one weight is nearly 1 and the slopes that reach the scores are tiny: softmax is saturated.
Switch to 1/√64: the box shrinks the scores, yet the scores get larger slopes, because softmax is no longer saturated.
The constant is not a knob: it never learns. It only rescales the slope that passes through it.

Inside an LLM · B · One transformer block

Causal mask No peeking ahead · masked scores get 0

During training the whole sentence is in memory, but a token must not read the tokens after it: at writing time they do not exist yet. The mask adds −∞ to every score of a future token. Its e becomes 0, its weight becomes 0, and going back it gets slope 0: the future token cannot be used, and learns nothing, from this position.

1

Walk the slope through it

The level before, the part, the level after

Three scores: an earlier token, this token, and a future token. The mask is the teal band. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Step back to score₃: slope 0. Its v₃ gets 0 too: a weight of 0 lets nothing through.
Switch the mask off: the future token gets the largest weight and starts receiving slope. That is the leak the mask prevents: the model would learn to copy the next word instead of predicting it.
The mask has no knob and no slope of its own to learn: it is a fixed + 0 or + (−∞).

Inside an LLM · B · One transformer block

Attention softmax Scores to weights · softmax without the p − y shortcut

Inside attention, softmax turns the scores into weights that add up to 1. No loss follows it, so the p − y shortcut does not apply here. Opened into small boxes (exp, add, divide), every rule stays simple. Each e reaches the loss by two roads, its own share and the total Z, and the two slopes add.

1

Walk the slope through it

The level before, the part, the level after

Three scores become three weights (teal band); the weighted sum of the values is the next level. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Open e₁: it feeds w₁ directly and Z, so its slope is the sum of two roads.
Z feeds all three divide boxes: raising Z shrinks every weight at once, and its slope collects all three effects.
The divide boxes' rule shows the one-line shortcut: each score's slope = its weight × (the slope at that weight − the weighted average of the slopes at all the weights).

Inside an LLM · B · One transformer block

Weighted sum of values Mix the values · earlier tokens learn from every later reader

Each token's attention output is its weights times the values of the tokens it reads, added. Here token 2 and token 3 both read token 1. Going back, v₁ collects slope from both readers: that is how a prompt token's value, and the weights and embedding row that made it, still get slope from every answer token that read it, even though the prompt is never graded.

1

Walk the slope through it

The level before, the part, the level after

Two readers, token 2 and token 3, each with its own loss; the two losses are averaged. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Open v₁: two multiply boxes read it, one per reader, and its slope is their sum.
Each multiply sends the value the slope × the weight, and the weight the slope × the value. Attention weights are not knobs, but their slope flows back through softmax and the scores to W_Q and W_K, which do learn.
The LM head is shared by both tokens: its knobs add the slopes of both.

Inside an LLM · B · One transformer block

Output projection W_O Back to the stream · one more matrix

The heads' outputs, side by side, go through one more weight matrix, W_O, which mixes them and maps them back to the stream's size. It is a matrix box like Q, K and V. Its output is added to the residual stream, so the slope it gets is exactly the slope of the stream after the add.

1

Walk the slope through it

The level before, the part, the level after

The attention output a goes through W_O (teal band) and is added to the stream x. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

The residual add copies the slope: o₁ and x₁ get exactly the same number.
Each weight's slope is the slope at its output × the attention number it multiplies.
a₁ feeds two multiply boxes, one per output, and adds their slopes.

Inside an LLM · B · One transformer block

Multi-head: split and concat Parallel heads · routing with slope × 1

Multi-head attention splits the vector into slices, one per head, lets every head attend on its own, then stacks the results back together. Splitting and stacking only move numbers, so their local slope is 1: each slope goes back to exactly the head and the slice it came from.

1

Walk the slope through it

The level before, the part, the level after

Two teal bands: the split, and the concat. Each head's attention is shortened to one multiply in between. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

The split and concat boxes pass the slope through unchanged: local slope 1.
The heads never swap slope: head 1's slope reaches only q₁, head 2's only q₂.
W_O is where the heads meet: it mixes them, so each head's output adds slope from both o₁ and o₂.

Inside an LLM · B · One transformer block

Residual add The highway · the slope goes around and through

Every block adds its result to its input instead of replacing it: r = x + block(x). An add box copies the slope to both inputs, so x gets the slope straight from the stream, plus whatever comes back through the block. Even when the block passes nothing back, the highway still does: that is a big reason very deep networks still train.

1

Walk the slope through it

The level before, the part, the level after

A block shortened to one weight and one ReLU per number, then the residual add (teal band). Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Number 2's gate is closed: the block sends 0 back, yet x₂ still gets the full slope through the highway.
Number 1's gate is open: x₁'s slope is the highway's slope plus the block's.
Without the highway, x₂ would get 0, and nothing below would learn through x₂ from this example.

Inside an LLM · B · One transformer block

MLP up projection Widen the vector · a bigger matrix

The MLP first widens each vector, 4 times its size in GPT-2, about 2.7 to 3.5 times in Llama, whose SwiGLU has two up matrices (here 2 → 3), so the activation has more numbers to work with. It is a matrix box: each input number feeds every wide number and collects their slopes, and each weight gets the slope at its output × its input.

1

Walk the slope through it

The level before, the part, the level after

The up projection is in the teal band; the GELU and the down projection are the next level. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

n₁ feeds three multiply boxes, one per wide number, and adds their slopes.
GELU in the next level has no hard cut-off, so even a negative wide number sends a little slope back.
In a real LLM the MLP's matrices (two in GPT-2, three with SwiGLU) hold about two thirds of each block's weights: most of the gradient's numbers live here.

Inside an LLM · B · One transformer block

Activation: ReLU, GELU, SwiGLU A non-linear gate · hard, smooth, or gated

Without a non-linear step, matrices in a row would collapse into one. ReLU is a hard gate: slope × 1 or × 0. GELU (GPT-2, BERT) is a smooth gate: its local slope fades toward 0 for very negative inputs (dipping slightly below 0) and rises to a little above 1, so a slightly negative input still passes some slope. SwiGLU (Llama) multiplies a smooth gate by a second projection, so the slope reaches both: the gate side gets the slope × the value, the value side gets the slope × SiLU(gate).

1

Walk the slope through it

The level before, the part, the level after

Switch the activation: the teal band redraws with ReLU, GELU or SwiGLU, and the down projection after it stays the same. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Switch to ReLU: u₂ is negative, so its gate closes and u₂ gets 0. Switch back to GELU: u₂ gets a small slope again.
GELU: the local slope of u₁ is a little above 1. A smooth gate can amplify the slope slightly.
SwiGLU: each multiply box sends the gate side the slope × the value, and the value side the slope × SiLU(gate).

Inside an LLM · B · One transformer block

MLP down projection + residual Narrow back · then the highway

The down projection maps the wide vector back to the stream's size, and the result is added to the stream. Same matrix rules as every projection, then the residual add, which copies the slope to the stream and to the MLP.

1

Walk the slope through it

The level before, the part, the level after

Three activations narrow back to two numbers (teal band), then join the stream x. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

The residual add copies: m₁ and x₁ get the same slope.
Each activation feeds two multiply boxes, one per output, and adds their slopes.
A weight's slope is its output's slope × the activation it multiplies: the small activation h₂ gives its weights small slopes.

Inside an LLM · C · Output side

Final LayerNorm Normalise before the LM head · mean and spread

The last block's output is normalised once more before the LM head. GPT-2 normalises here with LayerNorm, as in all its blocks; Llama uses RMSNorm here too (part 4.4). This lab shows LayerNorm: subtract the mean, divide by the spread, then a learned gain γ and shift β. Opened into small boxes, the mean is used again and again, so every input reaches the loss by many roads.

1

Walk the slope through it

The level before, the part, the level after

The last residual add is the level before; LayerNorm fills the teal band; the LM head reads its output. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Open μ: it feeds three subtract boxes, each with weight −1, and adds their slopes.
The shift β gets the slope unchanged (an add box); the gain γ gets the slope × the normalised number.
Add the slopes that reach r₁, r₂ and r₃: they come to about 0. LayerNorm ignores a shift of all the numbers by the same amount, so the slope cannot push them all the same way.

Inside an LLM · C · Output side

LM head and weight tying Score every word · one matrix used twice

The LM head scores every word of the vocabulary: one dot product per word. Many models tie it to the embedding table, so a word's score is the hidden vector · that word's embedding row. Then one table is used at both ends of the network, and a row used at both ends adds the slopes of both uses. (Here the head scores only Paris and Rome; a real tied head scores every row, so with tying every row gets a slope at every step.)

1

Walk the slope through it

The level before, the part, the level after

A token id and its embedding row, the network shortened to one weight per number, then the head (teal band). Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

With tying on and the prompt word Paris, E[Paris] is used by the lookup at the start and by the Paris score at the end. Open the Paris lookup: “used by 2 boxes”.
Switch the prompt word to France: E[Paris] is now used only by the head.
Switch tying off: the head gets its own matrix U, and the embedding rows learn only from the input side.

Inside an LLM · C · Output side

Softmax + cross-entropy, box by box The SFT loss · p − y comes out by itself

The first lab drew softmax + loss as one box with one rule, p − y. Here it is opened into its small boxes: exp, add, divide, ln and × −1. Step back through them and watch the slopes that reach the logits: they are exactly p − y, the magenta error slope.

1

Walk the slope through it

The level before, the part, the level after

The LM head makes three logits; the loss, opened into its small boxes, fills the teal band. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Only the right token's probability is computed: the loss needs nothing else. The other tokens act only through Z.
The right token's e reaches the loss by two roads, its own division and Z. Their slopes add, and after the exp box the logit gets exactly p − 1.
Switch the right token: the boxes redraw around the new target, and the error slopes move with it.

Inside an LLM · C · Output side

Loss mask over positions Grade only the answer · the prompt starts at 0

An SFT example has a prompt and an answer, and the model predicts the next token at every position. A position is graded when the token it must predict belongs to the answer, so the last prompt token, which predicts the first answer token, is graded too. The loss averages only the graded positions; the others get weight 0. Going back, a masked position's loss gets slope 0, so its logits and its last hidden vector get 0 too. The position is not cut off: in the layers below, the graded positions read its keys and values through attention (part 4.10), so its earlier vectors still get slope.

1

Walk the slope through it

The level before, the part, the level after

Three positions share one LM head; each has its own loss, and the masked average (teal band) decides what counts. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Step back to position 1's loss box (it predicts a prompt word): slope 0, and so is everything behind it in this lab.
The LM head is shared by every position: its knobs add the slopes of the graded positions only.
Switch to “every position”: position 1 now sends slope too, and each position's share drops to a third.

Inside an LLM · D · Post-training extras

LoRA adapter A small trainable side road · the big weight stays frozen

LoRA fine-tunes a model without changing its original weights. Next to a frozen matrix W it adds a thin side road: A squeezes the input down to a few numbers (here 1), B expands them back, and the result is added to W's output. The slope flows through both roads, but only A and B are trained.

1

Walk the slope through it

The level before, the part, the level after

Two roads from the same input: the frozen W and the side road A then B. Everything outside the side road is frozen. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

B starts at 0, so the side road adds nothing at first, and A's slope is exactly 0: no slope crosses a multiply by 0.
Press Train 1 step: only B moves, because A's slope was 0. Press again: B is no longer 0, and A starts learning too.
The table shows slopes for W and the LM head so you can compare, but a real framework never computes them. The slope still passes through W to the layers below. That is where LoRA saves memory: a frozen weight needs no slope and no AdamW averages; only A and B, a tiny share of the weights, carry them.

Inside an LLM · D · Post-training extras

Frozen reference model (DPO) A copy that never learns · the loss compares against it

DPO trains on pairs of answers, one chosen and one rejected. Its loss asks how much more the model now prefers the chosen answer than a frozen copy of the original model did. The reference's log-probabilities enter the margin as fixed numbers: the lab shows their slope, but trainers compute them with gradients off, and the frozen copy is never trained.

1

Walk the slope through it

The level before, the part, the level after

The policy's logits become log-probabilities, which meet the frozen reference's numbers in the margin, then σ, ln and × −1. Press Next box ◄ and Previous box ► to walk the slope in either direction, click any box to read its rule, and nudge it to check the slope.

2

What to look at

Open a reference box: it has a slope, but it is data from a frozen model, so nothing trains.
The slope that reaches each log-probability is one number, ±k, with k = β × (1 − σ(margin)) (the magenta box). It shrinks as the margin grows, so a pair stops pushing once the model prefers the chosen answer clearly more than the reference did.
Raise β: k grows, so each step pushes harder. Over a whole training run, a bigger β also keeps the model nearer the reference.

Cheat sheet

Every box: its job in 3 words, its rule What each box does going forward and what it sends back

The examples use the first lab's starting numbers, so you can find each one in that lab with Show all.

Box	In 3 words	Forward · left to right	Backward · what it sends left	In the lab
Input x	token as numbers	Given by the data.	Collects the slopes of every box it feeds.	x₁ feeds n1 and n2: 0.264 + 0.049 = 0.313
× weight	multiply by weight	input × weight	To the input: slope × weight. To the weight: slope × input.	n3→Paris: 0.975 × 0.50 slope −0.742 → input −0.371, weight −0.723
Sum + bias	add plus bias	Adds its inputs and its bias.	Copies the slope, unchanged, to every input and to the bias.	n3's sum: +0.330 goes to both weight boxes and to the bias
ReLU gate	keep only positives	Positive stays, negative becomes 0.	Open: passes the slope (× 1). Closed: sends 0.	n4's sum is −0.30: closed. −0.710 arrives, 0 passes
A shared number	add incoming slopes	One output feeds several boxes.	Its slope is the sum of theirs (rule 2).	n3 feeds Paris, Rome, Lyon: −0.371 + 0.748 − 0.047 = 0.330
Logit	raw token score	The output neuron's sum: one score per token.	Gets the error slope p − y: predicted minus target, so p − 1 for the right token and p for the others.	Paris: 0.258 − 1 = −0.742 Rome: 0.623 − 0 = +0.623
Softmax + SFT loss	probabilities, then −ln	Scores become probabilities; loss = −ln p(right token).	The walk starts here with slope 1, and leaves as the error slope p − y on each logit: the slope that makes the difference.	loss = −ln 0.258 = 1.355
Weighted sum	each input × share	A mean, a subtraction, a masked average.	Slope × that input's share: ½ for a mean of two, −1 for a subtraction, 0 for a masked position.	parts 4.2, 4.4, 4.17, 4.20, 4.22
Lookup	pick a row	Copies one row of a table.	Copies the slope into that row only. A row used twice adds both.	parts 4.1, 4.2, 4.3, 4.18
Multiply, two activations	both sides learn	a × b, both computed by the network.	a gets slope × b, and b gets slope × a.	parts 4.6, 4.9, 4.10
Multiply by a constant	scale it	x × c (1/√d, or −1).	Slope × c. The constant never learns.	parts 4.7, 4.19
Copy (split, concat)	route a number	Moves a number unchanged.	Slope × 1, back along the same route.	parts 4.12
Mask	+ 0 or −∞	Adds −∞ to a future token's score.	Its exp is 0, so 0 comes back.	parts 4.8
Square	square it	x².	Slope × 2x.	parts 4.4, 4.17
Square root	square root	√(x + ε).	Slope × 1 / (2 × its output).	parts 4.4, 4.17
Divide	divide	a ÷ b.	Top: slope × 1/b. Bottom: slope × (−a / b²).	parts 4.4, 4.9, 4.17
Exponential	e to the x	eˣ.	Slope × eˣ: its own output.	parts 4.9, 4.19
Natural log	take ln	ln x.	Slope × 1/x.	parts 4.19, 4.22
Sigmoid	squash to 0..1	σ(x).	Slope × σ × (1 − σ), at most ¼.	parts 4.22
GELU, SiLU	smooth gate	A ReLU with rounded corners.	Slope × a local slope that is 0.5 at 0, about 1 for large inputs, and fades toward 0 for negative ones, dipping slightly below 0.	parts 4.14, 4.15
The update (not a box)	step against slope	new = old − learning rate × slope, for every knob at once.		n3→Paris at rate 0.5: 0.50 − 0.5 × (−0.723) = 0.86

The same boxes in a large language model

A transformer adds new boxes: attention, with a softmax of its own, layer norm, which divides, and the embedding lookup. Each new box brings its own local slope, but the two rules that carry the slope through them never change. Plain SFT, the most common way to fine-tune a language model, is this walk at scale: one loss per answer token, the same walk back. Only the update is smarter: AdamW still steps each knob against its slope, but sizes each step from that knob's recent slopes.