Derivatives & Gradients
- Define the derivative as sensitivity: how much output moves per tiny input nudge
- Compute numerical derivatives with central differences and choose a sane step size h
- Compute partial derivatives of a two-variable function by hand and verify with NumPy
- Explain the gradient as the steepest-ascent direction and demonstrate it by sampling directions
- State when analytic beats numerical differentiation and why ML uses both (gradient checking)
| Spaced-rep warm-up: due flashcards (Days 50–52) | 10 min |
| ELI5 + tech read + 3B1B calculus ch. 1–3 | 25 min |
| Guided: derivative machine + steepest-ascent proof | 45 min |
| Practice: sensitivity clinic | 20 min |
| Project: gradcheck.py | 15 min |
| Quiz + flashcards | 10 min |
Builds on: Day 50 — Vectors — the gradient is one · Day 22 — Reading growth from numbers
You are in the shower fiddling with an unfamiliar dial. The question your hand is asking — "if I nudge this a tiny bit, how much does the temperature change, and which way?" — is a derivative. Nudge the dial a hair clockwise, temperature jumps 3 degrees: the derivative is +3 (steep! be careful!). Nudge it and almost nothing happens: derivative near 0 (a numb zone — turn harder). Temperature DROPS as you turn up: negative derivative (the plumber crossed the pipes). The derivative is not a formula first — it is a number attached to a point, measuring local sensitivity: output change per unit of tiny input change.
Now give the shower two dials — hot and cold. Each has its own sensitivity while the other is held still (partial derivatives), and the pair of them, written as a vector, is the gradient. Here is the one fact to carry forever: that little vector of sensitivities points exactly in the direction to turn BOTH dials at once for the fastest temperature increase. Every neural network ever trained is someone reading billions of these dials and nudging every single one a tiny step the OPPOSITE way — downhill, toward less error. Today you build the dial-reading machinery with your own hands; tomorrow chains dials together; Thursday you ride downhill.
Training IS derivatives: loss.backward() on Day 87 computes a gradient — one sensitivity number per parameter, billions of them — and every optimizer step nudges parameters against it. Engineers who internalize "gradient = local sensitivity" debug training runs instead of staring at them: exploding loss means huge sensitivities compounding (Day 89's clipping), a flat loss means numb dials (dead units, vanishing gradients). The numerical-vs-analytic distinction becomes gradient checking on Day 86 — the standard test for a hand-written backward pass — and "wiggle it and watch" remains your fallback tool for any black box, including prompt changes against an eval score.
Rolling downhill in fog — loss vs. weight, one step at a time
step 1 / 5The loss landscape: every possible weight value has a loss. We can't see the whole curve — only the slope where we stand.
Guided practice
Build a derivative machine and find its sweet spot
20 min- Run part 1: central-difference derivative of f(x) = x³ − 2x at x = 2.0. The analytic answer is 3x² − 2 = 10 exactly — confirm the estimate agrees to ~1e-10.
- Part 2 sweeps h from 1e-1 down to 1e-13 and prints the error at each. Find the U-shape: error falls as h shrinks (truncation fading) then RISES again (float round-off). Record your best h and worst h.
- Say the two failure modes out loud: big h = the function is not linear at that scale; tiny h = subtracting nearly equal floats leaves noise, then dividing by tiny h amplifies it.
- Swap in forward difference (part 3) and compare its best error to central's at the same h values — see the O(h) vs O(h²) gap as real digits lost.
The gradient really is the steepest way up
25 minThe function: f(x, y) = x² + 3xy + y² (an elliptical bowl-ish surface, tilted).
- Derive ∇f by hand on paper first: ∂f/∂x = 2x + 3y, ∂f/∂y = 3x + 2y. Evaluate at the point (1, 2). You should get [8, 7].
- Run part 1: the numerical gradient (central difference per coordinate) at (1, 2). Confirm it matches your hand answer to ~1e-9 — your first gradient check, the exact ritual Day 86 uses on backprop.
- Part 2 is the payoff: from (1, 2), step a fixed tiny distance in 24 compass directions and measure the actual increase in f. The printout marks the best direction found by brute force — confirm it aligns with ∇f/‖∇f‖ (cosine ≈ 1.0), that the worst is −∇f, and that the two zero-change directions are perpendicular to the gradient.
- Write the price-list sentence in your notes: change ≈ ∇f·u means the gradient prices every direction, and the dot product (Day 50!) is maximized along it.
On your own
Sensitivity clinic
20 minThree exercises, hand-then-verify. (1) For f(x) = 5x⁴ − x² + 7, derive f'(x), evaluate at x = −1, 0, 2, and verify each with your central-difference machine. (2) For g(x, y) = (x − 3)² + 2(y + 1)², derive ∇g, find the point where ∇g = [0, 0] BY HAND (set both partials to zero), and explain in one sentence what that point is on the surface. (3) The ML shape: for a single data point, loss(w) = (w·2.0 − 3.0)² — a prediction w×2 against target 3. Derive dloss/dw, find the w where it is zero, and check that your central-difference machine agrees at w = 0 and w = 5.
Constraints: paper first, code second; every analytic answer must be verified numerically to at least 1e-6.
Hints (only if stuck): (3) is the chain rule you will formalize tomorrow — write loss = u² with u = 2w − 3 and multiply the sensitivities (2u)·(2).
gradcheck.py — your derivative test kit
Build the tool you will still be using on Day 86: gradcheck.py with (1) numderiv(f, x, h=1e-5) — central difference, scalar; (2) numgrad(f, p, h=1e-5) — central-difference gradient for f taking an ndarray point of ANY dimension (loop coordinates, nudge one at a time); (3) check(f, analytic_grad, points) — compares analytic vs numerical at each point and reports max absolute and relative difference with a PASS/FAIL verdict at 1e-6. Include a pytest suite testing all three against the day's functions plus one 5-dimensional quadratic. Then gradients_notes.md: the U-curve table with your best-h reading, the steepest-ascent experiment result, and a five-line explanation of why numgrad's cost (2n evaluations for n dims) is fine for CHECKING but absurd for TRAINING a billion-parameter model (the sentence that makes Day 87's autograd feel inevitable).
Common mistakes & misconceptions
- Treating the derivative as "the formula" instead of a number at a point. Sensitivity is local: f'(2) = 10 says nothing about x = −3. Training reads sensitivities at the CURRENT parameters, every step.
- Making h as small as possible "for accuracy." Below ~1e-8 in float64, round-off dominates and estimates get WORSE. You mapped the U-curve; trust it.
- Using forward differences when central costs the same order and buys O(h²). One extra evaluation per coordinate for orders of magnitude better error is the best trade in numerics.
- Forgetting the other variables when taking a partial. ∂/∂x of 3xy is 3y, not 3 — y is frozen, not deleted. Hand-derive slowly; the numerical check catches this instantly.
- Thinking the gradient points "toward the minimum." It points steepest UPHILL from here — descent follows −∇f, and even that is only locally optimal, not aimed at the minimum (Day 55 shows the zig-zag consequences).
- Believing zero gradient means minimum. It means FLAT — minimum, maximum, or saddle. Day 54 shows why saddles, not minima, dominate high-dimensional landscapes.
Q1. f'(x) = −4 at the current x. What does a tiny step of +0.01 in x do to f?
Q2. Your central-difference estimate gets WORSE when you shrink h from 1e-6 to 1e-12. Why?
Q3. At point p, ∇f = [3, 4]. Which unit direction gives approximately ZERO change in f for a tiny step?
Go deeper — curated resources
- Directional derivatives, formally — D_u f = ∇f·u for unit u — the compass experiment as a theorem. MML book §5.2 proves the steepest-ascent claim you verified empirically.
- U-curve mapped; best h recorded with both failure modes explained
- Hand gradient at (1,2) matches numerical to 1e-9; compass experiment confirms steepest ascent
- Sensitivity clinic done paper-first, all answers verified numerically
- gradcheck.py committed with passing tests including the 5-D case
- Quiz ≥ 2/3
← Back: The gradient is a Day 50 vector, and "change ≈ ∇f·u" is a Day 50 dot product doing pricing work — steepest ascent is literally maximal alignment. The h sweep reused Day 22's habit of reading behavior from measured tables.
Forward →: Tomorrow the chain rule composes sensitivities through nested functions — the one rule backprop needs — and Day 55 turns −∇f into an algorithm with a step size. gradcheck.py returns verbatim on Day 86 to audit your hand-written backprop, and Day 87 replaces numgrad's 2n evaluations with autograd's single backward pass.
Unlocks: D54 Chain Rule & Optimization · D55 Gradient Descent Lab · D56 Week 8 Checkpoint: Linear Regression by Hand · D85 Neurons & Forward Pass