Day 56 · The math becomes code

Week 8 Checkpoint: Linear Regression by Hand

You will be able to

Reproduce the week's core machinery from memory: dot products, matmul-as-transformation, SVD truncation, gradients, the chain rule, GD regimes
Derive the MSE gradient for linear regression by hand and verify it with gradcheck
Train w and b with your own gradient descent and read the loss curves
Produce a learning-rate ablation table and defend the chosen rate
Write the intuition paragraph connecting this week to embeddings, backprop, and LoRA

Today's ~120 minutes

Closed-book recall drills over Days 50–55	25 min
Derive + gradcheck the MSE gradient	15 min
Explain-out-loud rehearsal	10 min
Build: train, ablate, and write the report	55 min
Cumulative quiz + flashcard deck drill	15 min

Builds on: Day 50 — Vectors & dot products · Day 53 — Derivatives & gradients · Day 54 — Chain rule · Day 55 — Gradient descent lab

The analogy

🧸 ELI5⚙ Tech

Seven days ago, vectors were arrows and derivatives were a shower dial. Today they conspire: you will teach a line to fit data, using nothing but this week's parts. The model is one dot product (w·x + b — Day 50). The loss is a bowl whose height is squared error. The gradient — derived by you, with Day 54's chain rule, checked by Day 53's checker — tells each parameter which way to nudge. And Day 55's descent loop rolls the parameters downhill until the line lies snugly through the cloud of points. Machine learning, complete, with no library and no mystery.

Why spend a checkpoint on the simplest model in the field? Because linear regression is the hydrogen atom of ML: every phenomenon you will ever debug — loss curves, learning rates, divergence, convergence — appears here in its purest form, small enough to inspect every number. The recall drills come first, as always: what you can rebuild from an empty file is what you actually own. When Day 71 shows you sklearn fitting this same model in one line, and Day 85 stacks these dot-product neurons into networks, you will know exactly what is inside the box — because you built the box.

Why this matters on the job

"Implement linear regression with gradient descent" is a real screening question for ML-adjacent roles — asked precisely because it composes vectors, gradients, and optimization into twenty lines and exposes whether the candidate owns the parts. It is also your first end-to-end training run: initialize → forward → loss → gradient → step → loss curve, the exact skeleton of every PyTorch loop you will write from Day 88 to the capstone. When a training run misbehaves in month six, you will debug it by shrinking the problem back toward today's transparent twenty lines — the hydrogen atom is also the diagnostic instrument.

Guided practice

guided 1

Closed-book recall drills — Week 8 edition

25 min

Editor closed. Write from memory, then diff against your notes and lab files; score each solid / shaky / gone.

Day 50: cosine similarity formula; typical cosine of random unit vectors in ℝ⁷⁶⁸ and why; the reason pipelines normalize embeddings.
Day 51: what a matrix's columns tell you; is AB = BA?; the shape story of (1000×768)@(768×512) and why batching = matmul matters.
Day 52: eigenvector in one line; SVD's three-move anatomy; the LoRA bet in one sentence.
Day 53: central-difference formula; why h has a floor; what direction ∇f points and what is perpendicular to it.
Day 54: compute by graph-walk: dL/dw for L = (w·x − y)² at w=2, x=3, y=5. (Answer: 2(6−5)·3 = 6. No peeking.)
Day 55: the three LR regimes; why canyons zig-zag; the momentum update from memory.

Shaky/gone items → revisit list; re-read ONLY those sections before building.

guided 2

Derive, check, and dry-run the regression gradient

15 min

On paper: derive ∂L/∂w and ∂L/∂b for MSE from scratch, showing the chain rule and the add-across-points step. Compare against the tech section only when done.
Generate the data (starter): 200 points, x ~ N(0, 2), y = 3x − 2 + noise(σ=1). Note the ground truth you will hunt: w=3, b=−2, and a loss floor near σ² = 1.
Implement mse_loss(w, b) and mse_grad(w, b) vectorized (two reductions, no loops), then verify the gradient with your Day 53 gradcheck at three points, e.g. (0,0), (3,−2), (−1, 4). All must pass at 1e-6 BEFORE training — never descend an unchecked gradient.
Sanity dry-run: at (w,b) = (0,0), state the SIGNS of both gradient components from the formulas alone (residuals are mostly negative where y is positive…), then confirm numerically.

🐍 python — editable, runs in your browser

import numpy as np

rng = np.random.default_rng(56)
N = 200
x = rng.normal(0, 2, N)
y = 3 * x - 2 + rng.normal(0, 1, N)      # ground truth: w=3, b=-2, noise sigma=1

def mse_loss(w, b):
    r = w * x + b - y                     # residuals, vectorized
    return float(np.mean(r ** 2))

def mse_grad(w, b):
    r = w * x + b - y
    return np.array([2 * np.mean(r * x),  # dL/dw: residual-input correlation
                     2 * np.mean(r)])     # dL/db: mean residual, doubled

# gradient check before trusting anything (Day 53's ritual)
def numgrad(f, p, h=1e-6):
    g = np.zeros_like(p)
    for i in range(len(p)):
        q1, q2 = p.copy(), p.copy()
        q1[i] += h; q2[i] -= h
        g[i] = (f(q1) - f(q2)) / (2 * h)
    return g

for point in ([0.0, 0.0], [3.0, -2.0], [-1.0, 4.0]):
    p = np.array(point)
    ana = mse_grad(*p)
    num = numgrad(lambda q: mse_loss(*q), p)
    ok = np.allclose(ana, num, atol=1e-5)
    print(f"grad check at {point}: analytic {ana.round(4)} "
          f"numerical {num.round(4)}  {'PASS' if ok else 'FAIL'}")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Explain it like a colleague is watching

10 min

Before the build, rehearse the understanding: set a 6-minute timer and explain out loud, as if pair-programming, (1) why the b-gradient is the mean residual (what the line does when it is uniformly too low), (2) why the w-gradient is a residual-input correlation (what tilting fixes), and (3) what the loss floor will be and why no amount of training goes below it.

Goal: no formula recited without its geometric reading. If any explanation stalls, that is your weakest link — reinforce it before training, because the write-up (and one day an interviewer) will ask exactly these three.

Ship before you stop

Linear regression by hand — the Week 8 build

Create linreg_by_hand.py + linreg_report.md. The build: import your Day 55 gd (or reimplement), train (w, b) from (0, 0) on the generated data, logging loss per step. Produce: (1) the loss-curve table (or matplotlib plot if you prefer) for the chosen η; (2) an LR ablation over η ∈ {0.001, 0.01, 0.05, 0.2, 0.5}: final loss, steps to within 1% of floor, verdict (creep/converged/oscillated/diverged); (3) recovered w, b vs ground truth 3, −2, and the loss floor vs σ² = 1; (4) a momentum bonus run (β=0.9) with its speedup; (5) the intuition paragraph — 8–12 sentences telling the whole story (dot product → residuals → chain-rule gradient → downhill loop → noise floor) with zero formulas, as if to a smart PM. Commit everything; Day 71 will rerun this exact fit with sklearn and your numbers must match.

Rubric — check what you completed (0/6)

Gradient passes gradcheck at three points before any training runRecovered parameters within ~0.1 of (3, −2); final loss near the σ²=1 floorAblation table covers all five rates with correct regime verdictsLoss curve for the chosen rate shows the steep-drop-then-floor shape, and the floor is explainedMomentum run recorded with speedup vs plain GDIntuition paragraph is formula-free, correct, and names the loss floor

Common mistakes & misconceptions

Training on an unchecked gradient. A sign error still descends — just to the wrong place, slowly and confusingly. Gradcheck first is a two-second insurance policy; make it a permanent habit (Day 86 depends on it).
Expecting loss → 0. The data contains noise σ²; the best possible line floors there. Chasing zero means chasing noise — your first encounter with the overfitting boundary (Day 76 formalizes it).
Reviewing by re-reading lab files instead of rebuilding from an empty buffer. The checkpoint's recall drills exist because retrieval, not recognition, is what sticks — same rule as Day 49.
Forgetting to average (or consistently sum) the loss and gradient. Mixing mean-loss with sum-gradient silently scales your effective learning rate by N — a classic "why does η=0.01 diverge here?" bug.
Sweeping η in tiny increments. Regimes live decades apart; scan powers of ten first (Day 55's rule), then refine. Your ablation table should SHOW the regimes, not five flavors of convergence.
Writing the intuition paragraph with formulas. The point is the FDE muscle: if you cannot tell the story without symbols, the understanding is still borrowed. Rewrite until a PM could repeat it.

Knowledge check

Q1. For MSE linear regression, ∂L/∂b = (2/N)·Σ(ŷᵢ − yᵢ). If the line currently sits uniformly BELOW the data, this gradient is…

Q2. Your regression loss curve plunges, then flattens at ≈ 1.02 and never improves. The injected noise had σ = 1. The right conclusion is…

Q3. The MSE loss surface in (w, b) is convex. What does this guarantee that will NOT hold for the neural networks of Phase 5?

Go deeper — curated resources

courseGoogle ML Crash Course — linear regression module (their loop vs yours) ↗25 min bookMathematics for Machine Learning — ch. 9 (linear regression, done formally) ↗30 min video3Blue1Brown — Essence of Linear Algebra (rewatch weak chapters from the drills) ↗20 min docsNumPy user guide — for the vectorized reductions ↗10 min

If you have a third hour

The closed-form ending — Convex + quadratic means calculus can skip the loop: setting the gradient to zero gives the normal equations, solvable directly (np.linalg.lstsq — which runs on Day 52's SVD). Fit your data with it and confirm it matches your GD answer; then note why closed forms die the moment models go nonlinear.

Done means

Recall drills scored; weak sections restudied from the revisit list
Gradient derived on paper and gradcheck-passed before training
Model recovers (3, −2) within tolerance; floor explained via σ²
Ablation table + loss curve + intuition paragraph committed
Cumulative quiz ≥ 2/3 (follow revisit pointers for misses)

How this connects

← Back: The whole week fit in twenty lines: Day 50's dot product is the model, Day 54's chain rule derived the gradient, Day 53's checker certified it, Day 55's loop trained it — and even Day 52 lurks in why the (w, b) valley is mildly canyon-shaped. Phase 2 shows too: the report habit comes from Day 49's design-doc-first discipline.

Forward →: Day 57 starts probability — the second pillar of ML math. Day 71 refits today's model with sklearn in one line (and your numbers must agree); Day 72 swaps the loss and gets classification; Day 85 stacks these neurons into networks; Day 86 automates today's hand gradient into backprop. The intuition paragraph is your first artifact in the Day 105 explainer lineage.

Unlocks: D57 Probability Fundamentals · D71 ML Framing & Linear Regression