MNIST Lab
- Train an MLP on MNIST to ≥ 97% test accuracy using your own trainer.py
- Build a small CNN, explain why convolutions beat flat pixels, and reach ≥ 98.5%
- Run three controlled ablations and record them in an experiment log
- Perform confusion-matrix error analysis and look at actual misclassified digits
- Ship a saved model plus a standalone inference script
| Spaced-rep warm-up + reread playbook.md | 10 min |
| Milestone 1: MLP baseline with sanity checks | 30 min |
| Milestone 2: CNN + three ablations | 35 min |
| Milestone 3: error analysis + predict.py | 25 min |
| Practice: one-change improvement attempt | 10 min |
| Quiz + flashcards + commit the lab | 10 min |
Builds on: Day 88 — Training loops & trainer.py · Day 89 — Training dynamics & the playbook · Day 75 — Confusion matrices & metrics
Every craft has its "hello, world," and deep learning's is teaching a machine to read handwritten digits. MNIST is 70,000 grayscale images, 28 by 28 pixels, each a scrawled 0–9 from real humans — census workers and high-schoolers of the 1990s. It is small enough to train on a laptop CPU in minutes and real enough to have every problem real data has: ambiguous scrawls where a 4 shades into a 9, sloppy 7s that look like 1s, and a model that will confidently get some of them wrong.
Today nothing is new — that's the point. You take the forward pass from Day 85, the loop from Day 88, and the playbook from Day 89, and you point them at real images for the first time. First a plain dial-stack (MLP) that treats the image as 784 unrelated numbers. Then a convolutional network, which stops shredding the picture into a list and instead slides small pattern-detectors across it — edges, curves, loops — the way your eye doesn't care WHERE in the frame a 7 appears, just that it's a 7. Then you do what real ML engineers spend most of their time doing: run experiments, log them honestly, and stare at the mistakes.
This is your first end-to-end deep learning deliverable — the artifact interviewers mean when they ask "have you actually trained a model?". The lab's real curriculum is the meta-skills: an experiment log with controlled one-variable changes (the discipline behind every serious eval you run from Day 134 on), error analysis by staring at real failures (Day 80's superpower applied to pixels), and an inference script with a clean contract — the exact seam where models meet production APIs on Day 148 when you containerize one.
Guided practice
Milestone 1 — MLP to 97%
30 min- Download MNIST via torchvision (starter). Note the Normalize transform and the train/test split arriving pre-defined.
- Sanity checks BEFORE training (playbook page one): print initial loss on one batch — must be ≈ 2.30 — then overfit a single batch of 64 to near-zero loss in ≤ 500 steps.
- Train the 784-256-10 MLP with your trainer.py: AdamW lr=1e-3, batch 64, 5 epochs, checkpointing best-by-val.
- Evaluate on the test set. Target ≥ 97.0%. If short, consult playbook.md — do NOT random-walk hyperparameters.
- Start experiments.md with this baseline row: config, epochs, final losses, test accuracy.
Milestone 2 — the CNN and the ablations
35 min- Build the CNN in the starter. Before training, predict its parameter count, then verify with sum(p.numel() for p in model.parameters()) — compare against the MLP's ~203k.
- Train 5 epochs, same recipe. Target ≥ 98.5% test accuracy.
- Run the three ablations, ONE change each, same seed, logging each as a row in experiments.md: a. No-normalization: drop the Normalize transform (both loaders!). Effect on convergence speed? b. Skinny MLP: hidden 256 → 32. How much accuracy does capacity buy? c. One-block CNN: delete the second conv block. Parameters vs accuracy?
- End each row with a one-sentence conclusion. No conclusion, no experiment.
Milestone 3 — error analysis + inference script
25 min- With the best CNN checkpoint, compute the 10×10 confusion matrix over the test set (loop with no_grad, tally predicted vs true).
- Identify the top confusion pair. Collect the indices of those misclassified images and display (or save as PNGs) at least 6 of them next to their true and predicted labels.
- For each: your verdict — model failure, genuinely ambiguous scrawl, or arguable label? Tally the three verdicts in experiments.md.
- Write
predict.py: loads the checkpoint, takes an image path (28×28 grayscale PNG) from argv, applies THE SAME normalization, prints the digit and the softmax confidence. Test it on 3 images you export from the test set. - Contract check: predict.py must not import trainer.py or the training script — inference stands alone.
On your own
Beat your own CNN
15 minSqueeze more out of the CNN with ONE additional playbook intervention of your choice — dropout before the head, weight decay, a third conv block, more epochs with early stopping, or an LR schedule. State your hypothesis in experiments.md BEFORE running, then the result after.
Constraints: one change; same seed; report test accuracy delta to two decimals.
Hint: at 98.5%+, gains are tenths of a percent — that is what real leaderboard grinding feels like, and why error analysis usually beats another hyperparameter run.
The MNIST lab report
Assemble the day into a mnist-lab/ folder in your practice repo: train_mnist.py (both architectures, driven by trainer.py), predict.py (standalone inference), experiments.md (baseline + 3 ablations + your practice hypothesis run, each with a conclusion sentence), the confusion matrix (text or image), 6+ misclassified images with verdicts, and the best checkpoint. A README paragraph states final numbers: MLP and CNN test accuracy, parameter counts of both, and your single most surprising finding. This lab gets refactored into the Day-82 project template on Day 91, so keep functions clean.
Common mistakes & misconceptions
- Adding softmax before CrossEntropyLoss. The loss applies log-softmax internally; doubling it squashes gradients and caps your accuracy mysteriously below target.
- Normalizing the training set but not the test set (or predict.py). Any transform mismatch between training and inference silently corrupts every prediction — the #1 deployment bug in vision.
- Changing two hyperparameters in one "ablation". You learn nothing attributable; one variable per run is the whole method.
- Reporting the last epoch's accuracy instead of the best checkpoint's. Your trainer already saves best-by-val — use it, and say which you report.
- Skipping the look-at-the-images step because the matrix "already shows it". The matrix says WHAT confuses; the pixels say WHY — and whether it is even fixable.
- Assuming CNN superiority means "more parameters". The CNN wins with FEWER parameters via sharing — check the counts you computed.
Q1. Why does a CNN beat an equally-sized MLP on images?
Q2. Your model outputs go through softmax, then into nn.CrossEntropyLoss. The result is…
Q3. predict.py skips the Normalize((0.1307,), (0.3081,)) transform the training pipeline used. Likely outcome?
Go deeper — curated resources
- Swap in FashionMNIST — Change one string in the dataset constructor and rerun everything. Same shapes, harder problem (~92% is respectable). Watching your identical pipeline score lower teaches more about dataset difficulty than any blog post.
- Both accuracy targets hit and recorded (MLP ≥ 97%, CNN ≥ 98.5%)
- experiments.md has baseline + 3 ablations + 1 hypothesis run, all with conclusions
- Confusion pair examined in pixels with verdicts tallied
- predict.py works standalone on exported PNGs
- Lab folder committed; quiz ≥ 2/3
← Back: Everything assembles: Day 85's layers, Day 88's loop and trainer.py, Day 89's playbook and sanity checks, Day 75's confusion matrix, and Day 80's stare-at-the-errors discipline.
Forward →: Day 91 refactors this lab into the Day-82 project template. The experiment-log habit becomes eval discipline on Day 134, predict.py's clean contract becomes the API seam Day 148 containerizes, and Day 99 repeats this arc — train, watch curves, analyze — on your own GPT.