Chain Rule & Optimization
- State the chain rule as sensitivity multiplication through composed functions
- Draw a computational graph and compute d(loss)/d(parameter) by walking it backwards
- Verify every hand-derived gradient with the Day 53 gradcheck kit
- Distinguish convex from non-convex loss surfaces and what each promises an optimizer
- Explain minima, maxima, and saddle points, and why saddles dominate in high dimensions
| Spaced-rep warm-up: due flashcards (Days 50–53) | 10 min |
| ELI5 + tech read + 3B1B chain rule chapter | 25 min |
| Guided: backward graph walk + compounding & terrain | 45 min |
| Practice: chain rule reps with gradcheck | 20 min |
| Project: backprop-on-paper dossier | 15 min |
| Quiz + flashcards | 10 min |
Builds on: Day 53 — Derivatives & gradients · Day 27 — Recursion — decomposing nested structure
Picture a bicycle drivetrain with several gears in a chain. You turn the pedal (a parameter); it turns gear A, which turns gear B, which turns the wheel (the loss). Question: how sensitive is the wheel to the pedal? Answer: multiply the gear ratios. If pedal→A triples the spin, A→B doubles it, and B→wheel halves it, then pedal→wheel is 3 × 2 × 0.5 = 3. That is the chain rule — the entire chain rule. Sensitivities of composed stages multiply.
Two consequences run all of deep learning. First, you can compute any long chain's sensitivity by walking backwards from the wheel, multiplying local ratios as you go — do it once and you know how EVERY gear in the chain affects the wheel. That backwards walk has a famous name: backpropagation (Day 86). Second, multiplication compounds: ten gears of ratio 2 give 1,024; ten gears of ratio 0.5 give about 0.001. Long chains make sensitivities explode or vanish — the exact disease deep networks suffer (Day 89 treats it). And once you can read sensitivities through any chain, optimization is just: nudge every pedal slightly the way that lowers the wheel — which works beautifully in a single smooth valley (convex) and gets interesting in mountain ranges with fake-flat mountain passes (saddle points) — the actual terrain of neural network training.
The chain rule is the single most important equation in modern AI: backprop is nothing but the chain rule executed efficiently over a computational graph, and on Day 86 you will implement it in ~60 lines and feel the whole mystery evaporate. The compounding intuition explains real production pathologies — vanishing and exploding gradients, why residual connections exist (Day 95), why gradient clipping is in every training script (Day 89). And the convex/non-convex distinction sets expectations you will defend in interviews: why linear regression has one right answer while neural nets have many good ones, and why "it converged" never means "it found THE minimum."
Gears in a chain — z = f(g(x)), and how blame multiplies backward
step 1 / 6A computational graph: x feeds gear g (square it), whose output u feeds gear f (triple it). Run x = 3 forward.
Guided practice
Walk the graph backwards, by hand and by code
25 minThe model: prediction = w·x + b, loss L = (w·x + b − y)², at x=2, y=5, w=1.5, b=0.5.
- On paper, draw the graph (w,x → u = w·x → v = u+b → r = v−y → L = r²) and run the forward pass, writing each node's value.
- Walk backwards on paper: dL/dr, then dL/dv, dL/db, dL/du, dL/dw, multiplying one local ratio per step. You should land on dL/dw = −6 and dL/db = −3.
- Run the starter — it does the same walk with printed narration. Confirm every intermediate matches your paper.
- The payoff move: check both against
numgradfrom your Day 53 gradcheck kit (already imported in the starter). Agreement to ~1e-6 = your first verified backprop. - Nudge test: bump w by +0.01 and recompute L. Confirm L changes by ≈ −6 × 0.01 — the sensitivity means what it says.
Compounding chains, and the terrain tour
20 min- Part 1 chains the SAME squashing function (logistic sigmoid) 1, 5, 10, and 20 times and computes d(output)/d(input) through the chain. Watch the gradient shrink toward zero as depth grows — sigmoid's local ratio maxes at 0.25, and 0.25²⁰ is dust. This is vanishing gradients, measured.
- Change the multiplier to a stage with ratio ~1.7 (part 1b) and watch 20 stages explode past 10⁴. Write both numbers in your notes next to the words "why depth was hard before residuals + careful init."
- Part 2 tours three 2-D surfaces at their critical points: a convex bowl, a double-well (two minima, a maximum between), and the classic saddle z = x² − y². For each, the script reports the gradient (≈0 at all three!) and what a small step in each axis direction does. State for each: min, max, or saddle — and why the gradient alone could not tell you.
- The high-dim punchline, out loud: a critical point is a MINIMUM only if it curves up in every direction; with a million directions, "up in all of them" is the rare case — most flat points are saddles.
On your own
Chain rule reps, graded by your own checker
20 minDerive each gradient BY HAND via a drawn graph, then verify with your gradcheck kit; all must pass at 1e-5. (1) L(w) = (sig(w·x) − y)² at x=1.5, y=0.8, w=0.3 — a one-neuron classifier's loss; reuse dsig. (2) L(w, b) = (w·x₁ + b − y₁)² + (w·x₂ + b − y₂)² with data (1, 2) and (2, 3.5) — TWO data points: note how the paths add (sum across paths, multiply along each). (3) f(x) = ln(1 + eˣ) at x = −2, 0, 3 — derive f'(x), simplify, and recognize the result (it is the sigmoid — softplus's sensitivity dial is sigmoid, a fact Day 85 reuses).
Constraints: show the graph and per-node locals for (1) and (2) in your notes; no symbolic shortcuts until your graph walk produced the answer once.
Hints (only if stuck): (2) is dL/dw = Σᵢ 2rᵢxᵢ with rᵢ the per-point residual — the shape of tomorrow's regression gradient.
The backprop-on-paper dossier
Create chainrule_dossier.md — the artifact you will re-read the night before Day 86. Contents: (1) the one-neuron graph drawn in ASCII with forward values and backward sensitivities at every node, from today's guided lab; (2) the multipath rule ("multiply along a path, add across paths") with your two-data-point exercise as the worked example; (3) the compounding table — sigmoid-chain gradients at depths 1/5/10/20 and the ratio-1.7 explosion — under the heading "why deep training needs help"; (4) the terrain field guide: bowl/well/saddle with the step-test evidence and the high-dimensional saddle argument in your own words; (5) a self-test: three fresh compositions (pick your own) derived by graph walk with gradcheck PASS lines pasted in. Commit alongside chain_lab.py containing all runnable code.
Common mistakes & misconceptions
- Memorizing "outer times inner" without evaluating the outer at the INNER VALUE. f'(g(x))·g'(x) — that f' is read at g(x), not at x. The computational graph makes this impossible to get wrong: each node differentiates at its own recorded input.
- Forgetting to ADD when a variable feeds multiple paths. w appearing in two loss terms contributes the SUM of both paths' products. Multiply along, add across — half of backprop bugs are a missing add.
- Believing vanishing gradients are a rare pathology. Chain twenty ratio-0.25 stages and it is arithmetic, not bad luck. Architectures (residuals), activations (ReLU), and init schemes exist specifically to keep local ratios near 1.
- Assuming gradient ≈ 0 means "found a minimum." Flat means flat: minimum, maximum, or (overwhelmingly, in high dimensions) saddle. Curvature — what steps in each direction do — makes the diagnosis.
- Expecting neural training to find THE global minimum. Non-convex losses have many minima; SGD finds A good one, and empirically that is fine. Convexity (regression) is the special case with a uniqueness guarantee, not the norm.
- Skipping the numerical check because the algebra "was easy." The nudge test and gradcheck take seconds and catch sign errors that would otherwise cost you an evening on Day 86. Verify every derivation today, while it is cheap.
Q1. y = f(g(x)) with g(2) = 5, g'(2) = 3, f'(5) = −2. What is dy/dx at x = 2?
Q2. A 30-stage chain has local sensitivity ~0.5 per stage. The end-to-end gradient is ~10⁻⁹. This phenomenon and its standard architectural fix are…
Q3. At a critical point of a million-parameter loss, the gradient is zero. Why is "saddle point" the safest bet?
Go deeper — curated resources
- Why backward beats forward for training — Forward-mode autodiff costs one pass per INPUT; reverse-mode costs one per OUTPUT. Loss functions have millions of inputs (parameters) and one output — reverse-mode wins by exactly that ratio. Day 87 makes this concrete in PyTorch.
- Paper backward walk matches code and numgrad at 1e-6; nudge test passes
- Vanishing and exploding chains measured and recorded
- All three practice gradients pass gradcheck at 1e-5
- Dossier committed with graph, multipath example, and terrain guide
- Quiz ≥ 2/3
← Back: Every backward step multiplied a Day 53 local sensitivity, and gradcheck.py graded your algebra. Decomposing nested computation into a graph is Day 27's recursion instinct pointed at calculus.
Forward →: Tomorrow, −∇L plus a step size becomes gradient descent and you will watch trajectories navigate exactly the terrains you toured. Day 86 scales today's backward walk into micrograd (Karpathy's lecture 1 is the same walk with code that grows); Day 89 treats vanishing/exploding with instruments; Day 95 shows residual connections doing chain-rule first aid inside transformers.
Unlocks: D55 Gradient Descent Lab · D56 Week 8 Checkpoint: Linear Regression by Hand · D86 Backpropagation from Scratch