Gradient Descent Lab
- Implement gradient descent from scratch and log full trajectories
- Demonstrate the three learning-rate regimes: creep, converge, oscillate/diverge
- Explain why ill-conditioned (elongated) valleys force zig-zagging and small steps
- Compare batch, mini-batch, and stochastic gradients on cost and noise
- Add momentum and show it damping oscillation and accelerating narrow valleys
| Spaced-rep warm-up: due flashcards (Days 53–54) | 10 min |
| ELI5 + tech read + gradient-descent visualizer | 20 min |
| Guided: LR regimes + canyon & momentum | 45 min |
| Practice: batch vs mini-batch vs SGD | 25 min |
| Project: the step-size field manual | 15 min |
| Quiz + flashcards | 10 min |
Builds on: Day 53 — Gradients — the downhill direction · Day 54 — Chain rule & loss terrains
You are on a foggy mountainside at night with a flashlight that shows only the ground at your feet. You cannot see the valley floor — but you can feel which way is downhill RIGHT HERE (the gradient), so you step that way and re-check. That loop — feel, step, repeat — is gradient descent, the algorithm training every neural network on Earth.
Everything interesting is in the step size. Tiny steps: you inch down safely but take all night (slow convergence). Confident medium steps: you stride to the bottom. Huge steps: you leap clear across the valley and land HIGHER on the far wall, leap back even higher — ping-ponging upward until you fly off the mountain (divergence). Worse, real valleys are not round bowls; they are long narrow canyons. Downhill-at-your-feet points mostly at the steep canyon WALL, barely along the gentle floor — so you zig-zag wall to wall while creeping forward. The fix is delightfully physical: be a heavy ball instead of a cautious hiker. Momentum remembers your recent direction; the wall-to-wall zigs cancel out while the persistent along-the-floor component accumulates. Today you implement all of it and watch the trajectories with your own eyes.
The learning rate is the single most important hyperparameter you will ever tune, and today you buy the intuition cheaply — on 2-D functions where you can SEE oscillation and divergence — instead of expensively, on a 4-hour training run that NaNs at 3 a.m. Loss-curve pathology reading (Day 89) is pattern-matching to what you produce deliberately today: plateau = LR too small or a saddle; wild spikes = too large or a bad batch. Mini-batch SGD is not a compromise but the industry default for deep learning — cheap steps, and noise that helps shake loose from saddles — and momentum-family optimizers (Adam included) are what torch.optim gives you on Day 88. Today is the lab where those defaults stop being folklore.
Rolling downhill in fog — loss vs. weight, one step at a time
step 1 / 5The loss landscape: every possible weight value has a loss. We can't see the whole curve — only the slope where we stand.
Guided practice
The three regimes, on a bowl you can predict
20 min- Read the starter's
gd()— ten lines, no magic. It logs every position so trajectories are inspectable. - Run on the round bowl L = x² + y² from (4, 3) with η ∈ {0.01, 0.1, 0.45, 0.9, 1.1}. For each, the script prints steps-to-converge (or DIVERGED) and the first few positions.
- Before each run, predict from the (1 − 2η) analysis which regime you will see; check yourself. Watch η = 0.9 overshoot back and forth with shrinking amplitude, and 1.1 ping-pong outward geometrically.
- Record your table: η, behavior, steps to reach loss < 1e-6. Note the asymmetric moral: too small wastes time; too large loses everything. The safe strategy in practice: start large-ish, decay.
The canyon, and the heavy ball that tames it
25 min- Switch to the canyon L = x² + 10y², start (9, 1). Gradient: [2x, 20y]. First find the divergence threshold empirically: try η = 0.05, 0.09, 0.11. (The steep axis multiplies by (1 − 20η) — instability arrives at η = 0.1, far below the bowl's 1.0.)
- At the safe η = 0.09, print the first 12 positions and LOOK at them: y flips sign nearly every step (wall-to-wall zig-zag) while x grinds down slowly. Count steps to loss < 1e-6. This is ill-conditioning: the steep direction sets the speed limit; the gentle direction pays it.
- Now run
gd_momentumwith the same η = 0.09, β = 0.9. Compare steps-to-converge (expect several-fold fewer) and watch the y-column: oscillations damp as opposing gradients cancel in the velocity while x-progress compounds. - Sweep β ∈ {0, 0.5, 0.9, 0.99} and tabulate. Note that β = 0.99 overshoots and rings before settling — momentum has its own too-much regime.
- Notes: two sentences on why real training (condition numbers in the thousands) makes momentum-family optimizers the default, not a luxury.
On your own
Batch vs mini-batch vs stochastic, measured
25 minFit the 1-parameter model y ≈ w·x to 1,000 synthetic points (x from N(0,1), y = 3x + noise·0.5) by minimizing mean squared error. The full-data gradient is dL/dw = mean(2·(w·xᵢ − yᵢ)·xᵢ); a batch's gradient is the same mean over the batch only.
Run three descents from w = 0 at η = 0.05, an "epoch budget" of 30 passes over the data: (1) batch GD — one exact step per epoch; (2) mini-batch, size 32 — shuffled each epoch; (3) SGD, batch size 1. For each, log w after every epoch and produce: a table of w-trajectories, gradient EVALUATIONS per epoch, and the final |w − 3|.
Answer in your notes: which method takes the most steps but the fewest data passes per step? Why does the SGD trajectory never quite settle (look at its last five epochs)? Which would you pick if the dataset were 10 million points, and why does the noise become a feature at saddle points (Day 54)?
Hints (only if stuck): all three see the same data per epoch — the difference is exactness per step vs steps per epoch; SGD's endpoint jitters in a band whose width scales with η.
The step-size field manual
Create gd_lab.py + gd_field_manual.md in your practice repo. The script: your gd/gd_momentum implementations plus the three experiments (regimes table, canyon + momentum sweep, batch-variant comparison), each behind a function, all reproducible with one seed. The manual: (1) the regimes table with the (1 − 2η) explanation in your own words; (2) the canyon story — why steep directions set the speed limit, with your zig-zag printout as evidence; (3) the momentum sweep table and the "average of last ~10 gradients" reading of β = 0.9; (4) the batch-variant table with the cost/noise trade; (5) a ten-line "symptom → suspect" cheat sheet (loss flat → LR tiny or saddle; loss spikes then recovers → borderline LR or noisy batch; loss NaN → diverged, cut LR 10×...). Tomorrow's checkpoint uses gd_lab.py directly on real regression — write it importable.
Common mistakes & misconceptions
- Tuning the learning rate by tiny increments from a bad start. The regimes differ by orders of magnitude — search η in powers of 10 first (0.001, 0.01, 0.1), then refine within the stable decade.
- Reading oscillation as "noise" and averaging it away mentally. Systematic overshoot (sign-flipping coordinates, loss bouncing) is a too-large step size announcing itself — cut η, do not squint.
- Expecting the bowl's safe η to transfer to the canyon. Stability is set by the STEEPEST direction; ill-conditioned losses demand steps sized for their worst axis. That is why the same η that cruised on one problem diverges on another.
- Treating SGD noise as pure downside. The wander costs precision near the minimum but shakes trajectories off saddles and plateaus — part of why deep learning works. Decay η late to trade noise back for precision.
- Cranking momentum toward 1.0 for "more speed." β = 0.99 averages ~100 gradients: the ball gets so heavy it overshoots and rings. 0.9 is the default for a reason; tune it like a damper, not a throttle.
- Judging convergence by the last loss value alone. Log trajectories. A loss of 0.01 could be converged (gradient ~0) or mid-oscillation; the path — like your printed tables — tells you which.
Q1. On L = x² + y², gradient descent with η = 1.1 produces iterates that get FARTHER from the minimum each step. Why?
Q2. In the canyon L = x² + 10y², why does plain GD zig-zag?
Q3. Mini-batch SGD is the deep-learning default over full-batch GD mainly because…
Go deeper — curated resources
- Why the canyon is eigenvalues in disguise — The Hessian of x² + 10y² has eigenvalues 2 and 20; stability requires η < 2/λ_max and convergence speed is set by λ_min — condition number κ = λ_max/λ_min predicts the pain. Day 52's decomposition, running the optimizer's speed limit.
- All five LR regimes produced and matched to prediction
- Canyon zig-zag observed in printed trajectories; momentum ≥ 3× speedup measured
- Three batch variants compared with cost and final-error table
- Field manual committed with the symptom cheat sheet
- Quiz ≥ 2/3
← Back: Every step is Day 53's −∇L pointed downhill, marching over Day 54's terrains — and the canyon's steepness ratio is secretly Day 52's eigenvalue story (curvature eigenvalues set the speed limit). The batching economics are Day 51's matmul argument.
Forward →: Tomorrow gd_lab.py trains a real model: linear regression by hand, closing the math week. Day 71 reveals sklearn doing the same fit in one line; Day 88 hands you torch.optim.SGD(momentum=0.9) and Adam, which you can now read ingredient by ingredient; Day 89's loss-curve clinic is today's symptom sheet at production scale.
Unlocks: D56 Week 8 Checkpoint: Linear Regression by Hand · D71 ML Framing & Linear Regression · D86 Backpropagation from Scratch · D88 Training Loops & Data