Day 86 · Blame flows backwards

Backpropagation from Scratch

You will be able to

Explain backprop as the chain rule applied over a computational graph, node by node
Derive the local gradient rules for +, ×, and tanh and apply them by hand
Build a complete micrograd-style scalar autograd engine in ~60 lines of pure Python
Verify analytic gradients against numerical gradients (gradient checking)
Train a tiny neuron with your own engine and watch the loss fall

Today's ~120 minutes

Spaced-rep warm-up: Day 54 chain-rule cards + Day 85 recall	10 min
ELI5 + tech read; Karpathy micrograd video (first hour, 1.5x)	30 min
Guided: build the engine, gradient-check it, train a neuron	55 min
Practice: extend with relu and pow	15 min
Quiz + flashcards	10 min

Builds on: Day 54 — Chain rule & computational graphs · Day 55 — Gradient descent lab · Day 85 — Neurons & the forward pass

The analogy

🧸 ELI5⚙ Tech

A restaurant serves a bad dish and the review comes in: 2 stars. Who gets the blame? The head chef doesn't yell at everyone equally — blame flows backwards along the assembly line. The plating station passes most of the blame to the sauce station ("the dish was fine until the sauce"), the sauce station splits it between the salt and the cream, and by the end every single station knows exactly how much IT contributed to the bad review — and, crucially, in which direction to adjust. Too much salt gets "use less"; undercooked base gets "cook longer."

Backpropagation is precisely this blame audit, run on the network you built yesterday. The loss is the bad review. Each operation in the forward pass (every add, every multiply, every tanh) knows its own tiny local rule for passing blame to its inputs. Chain those local rules from the loss backwards to every weight and you get, for each dial on the mixing desk, a number: "turn me this way, this much, and the review improves." Gradient descent (Day 55) then turns every dial a little. Repeat ten thousand times and the network has learned. Today you build the blame-routing machinery yourself, in sixty lines of Python, and it is the same machinery inside PyTorch.

Why this matters on the job

"Just use autograd" is fine until a gradient silently becomes zero, a loss plateaus, or a customer asks why fine-tuning diverges — then the engineers who understand blame-flow debug in minutes while others tweak randomly for days. This lesson is also the single highest-leverage interview topic in deep learning: "explain backprop" appears in nearly every ML-adjacent loop, and having WRITTEN one beats having read about one. Day 87's PyTorch will feel like a familiar machine with a nicer case, because you built the engine.

Watch it happen

Blame flows backwards — the same net, gradients running right to left

step 1 / 5

Forward is done: prediction 0.51, truth 1.0, loss 0.24. Training's question: how much is EACH dial (weight) to blame? Backprop answers by walking the graph in reverse.loss = (0.51 − 1.0)² = 0.24

🎯 Predict modespeed1×

Guided practice

guided 1

Build the Value engine (the whole thing)

25 min

Create micrograd_mine.py (or a browser Python cell) and type — do not paste — the Value class below. Typing it is the lesson.
Reproduce this tiny graph: a = Value(2.0), b = Value(-3.0), c = Value(10.0), d = a*b + c, L = d * Value(-2.0).
Before calling backward, predict on paper: dL/dd = -2; dL/dc = -2 (addition routes); dL/da = b · dL/dd = -3 · -2 = 6; dL/db = a · dL/dd = 2 · -2 = -4.
Call L.backward() and confirm all four predictions.
Add a tanh to the chain and confirm blame shrinks by (1 - tanh²) at that node.

🐍 python — editable, runs in your browser

import math

class Value:
    """A scalar with a gradient, recording the graph that produced it."""

def __init__(self, data, _children=()):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)

def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other))
        def _backward():
            self.grad  += 1.0 * out.grad   # addition routes blame through
            other.grad += 1.0 * out.grad
        out._backward = _backward
        return out

def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other))
        def _backward():
            self.grad  += other.data * out.grad  # scaled by the OTHER input
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,))
        def _backward():
            self.grad += (1 - t * t) * out.grad  # dies near saturation
        out._backward = _backward
        return out

def __neg__(self): return self * -1
    def __sub__(self, other): return self + (-other)
    def __radd__(self, other): return self + other
    def __rmul__(self, other): return self * other

def backward(self):
        topo, visited = [], set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)
        self.grad = 1.0                      # dL/dL = 1
        for node in reversed(topo):
            node._backward()

a = Value(2.0); b = Value(-3.0); c = Value(10.0)
d = a * b + c
L = d * Value(-2.0)
L.backward()
print('dL/dd =', d.grad, ' dL/dc =', c.grad)   # -2, -2
print('dL/da =', a.grad, ' dL/db =', b.grad)   # 6, -4

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Gradient-check your engine

15 min

Write numerical_grad(f, x, h=1e-6) returning (f(x+h) - f(x-h)) / (2*h) for a plain-float function f.
Check dL/da from exercise 1: define f(a_val) that rebuilds the whole expression with a = Value(a_val) and returns L.data. Compare against a.grad = 6.
Repeat for b and c. All three must agree to ~1e-6.
Now sabotage the engine: change addition's backward rule to route 0.5 instead of 1.0 and rerun the check. Watch the checker catch the bug — this is exactly how framework authors test autograd.
Restore the correct rule.

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 3

A neuron learns with your engine

15 min

Build one neuron from Value objects: two weights, one bias, tanh activation.
Training set: the four XOR-ish points below with targets in {-1, 1} (a single neuron can only fit the linearly separable subset — that is part of the lesson).
Loop 50 times: forward on all points, squared-error loss, zero all grads (set .grad = 0.0 on every parameter — forgetting this is THE classic bug), backward, then nudge each parameter by -0.1 * grad.
Print the loss every 10 steps. It must fall.
In one sentence: why must gradients be zeroed each step, given the += in your backward rules?

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Extend the engine

20 min

Add two operations to your Value class: relu() and __pow__(self, k) for a constant float k (enough to write x**2 for losses). Then gradient-check both with your numerical checker on at least two input values each — including x = -1.5 for relu, where the gradient must be exactly 0.

Constraints: follow the same pattern (compute out, define _backward with +=, attach). No peeking at the micrograd repo until your checks pass.

Hints: d(relu)/dz is 1 if z > 0 else 0; d(x^k)/dx = k·x^(k-1). For relu at exactly 0, pick either convention — note which you chose and why it rarely matters in practice.

Ship before you stop

micrograd_mine — your autograd engine, tested

Promote today's work into your practice repo as micrograd_mine.py plus test_micrograd.py: the Value class with add/mul/tanh/relu/pow/neg/sub, a Neuron class (weights, bias, tanh) built on it, and pytest tests that (a) reproduce the hand-derived gradients from guided 1, (b) gradient-check every operation numerically, and (c) assert the guided-3 training loop reaches loss < 0.1 in 50 steps with seed 0. This is the artifact Day 87 rebuilds in PyTorch and Day 91 asks you to re-derive from memory — invest in it.

Rubric — check what you completed (0/6)

Engine supports add, mul, tanh, relu, pow, neg, sub with correct += accumulationbackward() uses an explicit topological sort, not recursion luckAll gradient checks pass within 1e-4 relative toleranceTraining test reaches loss < 0.1 deterministically (seeded)Sabotage test exists: a deliberately wrong rule is caught by the checkerCommitted with tests green (pytest exit 0)

Common mistakes & misconceptions

Using = instead of += when writing gradients. A value feeding two paths receives blame from both; overwriting silently drops one path and the bug is invisible on simple graphs.
Forgetting to zero gradients between steps. Because backward accumulates, step 2 trains on step 1's gradients plus its own — loss jitters or explodes.
Calling _backward in arbitrary order. A node's rule needs its COMPLETE output gradient first; that is what the reverse topological order guarantees.
Believing backprop finds "the global answer". It computes exact gradients of the loss at the current point; where descent then leads is Day 55's non-convex reality.
Confusing the numerical gradient's role: it is a TEST harness, not a training method — it costs one forward pass per parameter per step.
Thinking tanh saturation is exotic trivia. A saturated unit (output near ±1) passes ~0 blame: whole layers can stop learning. Day 89's tricks exist for this.

Knowledge check

Q1. In c = a · b with a = 2, b = -3, and dL/dc = 4, what is dL/da?

Q2. A Value feeds into two different downstream operations. Its gradient must be…

Q3. Your analytic gradient says 0.5001 and the numerical gradient says 0.4999. The right conclusion is…

Go deeper — curated resources

videoKarpathy — The spelled-out intro to backpropagation: building micrograd ↗2 h 25 min (watch through the Value class; finish across the week)video3Blue1Brown — What is backpropagation really doing? (ch. 3–4) ↗25 min courseKarpathy — Zero to Hero course page ↗5 min bookDeep Learning (Goodfellow) — ch. 6.5, Back-Propagation ↗25 min

If you have a third hour

Read the real micrograd (karpathy/micrograd on GitHub) — Under 100 lines for the engine. Diff it against yours: the main additions are __pow__ generality and nn.py's Neuron/Layer/MLP classes — which you now understand completely.

Done means

Value engine built BY TYPING and all four hand-predicted gradients confirmed
Gradient checker passes on a, b, c and catches the sabotaged rule
Neuron training loop reaches falling loss with grads zeroed each step
micrograd_mine.py + tests committed, pytest green
Quiz ≥ 2/3

How this connects

← Back: This is Day 54's chain rule made executable and Day 55's gradient descent given its gradients; the graph you differentiate is exactly Day 85's forward pass.

Forward →: Day 87: PyTorch's autograd is this engine with tensors, C++ speed, and a thousand ops — you will rebuild today in 20 lines. Day 97's GPT trains by this exact mechanism, and "explain backprop from memory" is Day 91's centerpiece drill.

Unlocks: D87 PyTorch — Tensors & Autograd · D91 Week 13 Checkpoint — The First Neural Check · D93 word2vec Lab