Day 88 · The practice schedule

Training Loops & Data

You will be able to

Write the canonical PyTorch training loop from memory: forward, loss, zero_grad, backward, step
Wrap data in Dataset and DataLoader with batching and shuffling, and explain why each matters
Switch correctly between model.train() and model.eval(), and pair eval with no_grad
Save and load checkpoints (model + optimizer state) and resume training
Structure a train/validate epoch loop that records losses for later diagnosis

Today's ~120 minutes

Spaced-rep warm-up: Day 87 cards + recite the five beats	10 min
ELI5 + tech read; Datasets/DataLoaders + Optimization tutorial pages	25 min
Guided: canonical loop, checkpoint/resume, custom Dataset	50 min
Practice + project: extract and commit trainer.py	25 min
Quiz + flashcards	10 min

Builds on: Day 87 — Tensors, autograd & nn.Module · Day 55 — Batch vs mini-batch gradient descent · Day 76 — Validation discipline

The analogy

🧸 ELI5⚙ Tech

A music teacher doesn't hand a student the entire songbook and say "practice." She builds a practice schedule: shuffle the pieces so the student doesn't just memorize the order, work through them in small sets so each session gives quick feedback, and after every set, correct what went wrong. Once a week there's a mock recital — no correcting allowed, just honest measurement of how the pieces sound cold. And she keeps a notebook, so if lessons stop for a month, practice resumes exactly where it left off instead of starting over.

That is the training loop. The DataLoader is the schedule-maker: it shuffles the dataset each epoch and deals it out in mini-batches. Each batch gets the five-beat correction ritual — predict, score, clear old notes, assign blame, adjust. The mock recital is the validation pass: model.eval() plus no_grad, measurement without learning. The notebook is the checkpoint file: model weights and optimizer state saved to disk, so a crash, a preemption, or a curious teammate can pick up mid-song. You will write this loop hundreds of times in your career; today you learn it so cold that on Day 99 you type it without thinking while your GPT trains.

Why this matters on the job

The loop is the assembly line of all deep learning: MNIST (Day 90), your GPT (Day 99), and LoRA fine-tunes (Day 128) differ only in the model and data plugged into it. Production incidents live here too: a team forgetting model.eval() ships a model whose dropout is still on, quietly degrading every prediction; a missing optimizer state in a checkpoint makes "resume training" silently diverge. As an FDE you will read customers' training scripts — recognizing a malformed loop at a glance is a real diagnostic skill.

Guided practice

guided 1

The canonical loop on a real (small) problem

25 min

(Local run.) The starter trains your Day-87 MLP on a synthetic two-moons-style dataset: 1,000 points, two interleaved classes — a problem a linear model cannot solve.
Read the loop line by line and annotate each of the five beats with a comment IN YOUR OWN WORDS.
Run it. Training loss should fall below 0.3 within 20 epochs; validation accuracy should exceed 95%.
Now break beat 3: comment out zero_grad and rerun. Describe the loss curve in one sentence (Day 86 told you why).
Restore it, then set shuffle=False AND sort the data by label before loading. Watch the loss oscillate as each epoch sees a long run of class 0 then class 1 — this is why we shuffle.

🐍 python — editable, runs in your browser

import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader

torch.manual_seed(0)

# synthetic "two arcs" dataset
n = 500
t = torch.rand(n) * 3.14159
x0 = torch.stack([t.cos(), t.sin()], dim=1) + torch.randn(n, 2) * 0.1
x1 = torch.stack([1 - t.cos(), 0.5 - t.sin()], dim=1) + torch.randn(n, 2) * 0.1
X = torch.cat([x0, x1]);  y = torch.cat([torch.zeros(n), torch.ones(n)]).long()

perm = torch.randperm(2 * n)          # split AFTER shuffling the pool
X, y = X[perm], y[perm]
train_ds = TensorDataset(X[:800], y[:800])
val_ds   = TensorDataset(X[800:], y[800:])
train_loader = DataLoader(train_ds, batch_size=32, shuffle=True)
val_loader   = DataLoader(val_ds, batch_size=64, shuffle=False)

model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-2)

for epoch in range(20):
    model.train()
    total = 0.0
    for xb, yb in train_loader:
        pred = model(xb)              # 1 forward
        loss = loss_fn(pred, yb)      # 2 score
        optimizer.zero_grad()         # 3 clear old blame
        loss.backward()               # 4 assign blame
        optimizer.step()              # 5 adjust
        total += loss.item() * len(xb)

model.eval()
    correct = 0
    with torch.no_grad():             # eval mode AND no_grad
        for xb, yb in val_loader:
            correct += (model(xb).argmax(1) == yb).sum().item()
    print(f'epoch {epoch:2d}  train_loss {total/800:.3f}  val_acc {correct/200:.3f}')

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Checkpoint, kill, resume

15 min

Add checkpointing to exercise 1: every epoch save {'model', 'optim', 'epoch'} to ckpt_latest.pt, and to ckpt_best.pt when validation accuracy improves.
Train 10 epochs, then stop the script (simulate a crash).
Write resume.py: rebuild model and optimizer, load the checkpoint, and continue from the saved epoch. Confirm the first resumed epoch's loss continues the old curve rather than jumping up.
Sabotage test: resume loading ONLY the model weights (fresh optimizer). Compare the first resumed epoch — with AdamW the loss typically blips upward because momentum statistics were lost. Note this in your log.

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 3

Write a custom Dataset

10 min

Implement class CSVPoints(torch.utils.data.Dataset) that loads a CSV of x1,x2,label rows in __init__, returns (features_tensor, label_tensor) from __getitem__, and its row count from __len__.
Generate a 200-row CSV from exercise 1's data with NumPy, load it through your Dataset + a DataLoader, and verify one batch has shapes (32, 2) and (32,).
One-sentence answer in your notes: why does the DataLoader need __len__?

On your own

Extract your reusable trainer

20 min

Refactor exercise 1 into train_model(model, train_loader, val_loader, loss_fn, optimizer, epochs, ckpt_path) in a file trainer.py: runs the epoch loop, records per-epoch train loss and val metric into a history dict, checkpoints latest+best, and returns the history. Prove reusability by training TWO different models with it (the 16-unit MLP and a 64-unit one) and printing both histories.

Constraints: no globals; the function must not reference the dataset shape anywhere; eval must use eval mode + no_grad.

Hints: pass a metric_fn(pred, yb) instead of hardcoding accuracy; that keeps regression tasks possible on Day 90's ablations.

Ship before you stop

trainer.py — the loop you will reuse forever

Commit trainer.py (the reusable train_model function plus a checkpoint save/load pair) and train_arcs.py (the two-arcs experiment driving it) to your practice repo. Include the two deliberate-failure notes as comments: what the loss did without zero_grad, and what happened on resume without optimizer state. Requirement: running python train_arcs.py end-to-end trains, checkpoints, and prints a final validation accuracy ≥ 0.95. Day 90's MNIST lab imports train_model unchanged — if it needs edits there, today's abstraction leaked.

Rubric — check what you completed (0/6)

Five-beat loop correct, with eval mode + no_grad on the validation pathtrain_model works unchanged for two different architecturesCheckpoints store model AND optimizer state; resume continues the loss curveHistory dict captures per-epoch train loss and val metricBoth deliberate-failure experiments documented in commentsFinal val accuracy ≥ 0.95 on the arcs dataset

Common mistakes & misconceptions

Forgetting model.eval() for validation. Dropout keeps dropping and batch norm keeps updating — metrics are silently wrong and the "bug" reproduces nowhere else.
Thinking no_grad and eval() are interchangeable. One stops graph recording, the other changes layer behavior; validation needs both.
Shuffling the validation loader or, worse, NOT shuffling training data sorted by class — the first makes runs incomparable, the second makes each epoch a sequence of biased gradients.
Checkpointing only the model. Adam/AdamW carry per-parameter state; resuming without it makes the first epochs stumble and corrupts learning-curve comparisons.
Computing the epoch loss as a mean of batch means with uneven batch sizes. Weight by batch length (as the starter does) or the last small batch skews the number.
Calling loss.item() inside the graph-hot path unnecessarily is fine — but logging the raw tensor keeps the whole graph alive in your history list. Store floats, not tensors.

Knowledge check

Q1. Correct order of the five beats?

Q2. Validation numbers look great in the notebook but the deployed model performs worse. The loop-related suspect is…

Q3. Why restore optimizer.state_dict() when resuming AdamW training?

Go deeper — curated resources

docsPyTorch — Learn the Basics (Datasets & DataLoaders, Optimization pages) ↗40 min docsPyTorch Tutorials hub (Quickstart end-to-end) ↗20 min coursefast.ai Practical Deep Learning — training mechanics lessons ↗30 min bookDive into Deep Learning — the training loop chapters ↗20 min

If you have a third hour

Gradient accumulation — When the batch you want exceeds memory: run K forward/backwards without zeroing (exploiting += !), then one step. Effective batch = K × loader batch. You will see this in every LLM training script, including nanoGPT on Day 99.

Done means

Canonical loop annotated and run; both sabotage experiments observed and explained
Checkpoint → kill → resume demonstrated with a continuous loss curve
trainer.py committed and reused across two architectures
Arcs validation accuracy ≥ 0.95
Quiz ≥ 2/3

How this connects

← Back: Beat 3–5 are Day 86's zero/backward/update ritual with an optimizer object; mini-batching is Day 55's stochastic gradient trade-off; the untouched-validation-set discipline is Day 76 wearing PyTorch clothes.

Forward →: Day 89 teaches you to READ the loss histories this loop produces. Day 90 points trainer.py at MNIST. Day 99 trains your GPT with this identical loop, and Day 128's LoRA fine-tune is this loop with most parameters frozen.

Unlocks: D89 Training Dynamics · D90 MNIST Lab · D91 Week 13 Checkpoint — The First Neural Check · D97 Tiny GPT Lab I — Build It