Training Dynamics
- Diagnose underfitting, overfitting, and instability from train/validation loss curves alone
- Apply the regularization toolkit — weight decay, dropout, early stopping — and say what each trades away
- Explain what batch norm does at a gist level and why train/eval mode matters for it
- Choose learning rates, schedules, and warmup with a principled starting recipe
- Use gradient clipping to tame exploding gradients and know when it is a band-aid
| Spaced-rep warm-up: Days 86–88 cards | 10 min |
| ELI5 + tech read; study the loss-curves visualizer signatures | 20 min |
| Guided: manufacture pathologies, treat the overfitter, rescue the hot oven | 55 min |
| Practice + project: diagnose() and playbook.md | 25 min |
| Quiz + flashcards | 10 min |
Builds on: Day 88 — Training loops & checkpoints · Day 76 — Bias/variance & regularization · Day 55 — Learning rates & divergence
A baker learns more from watching through the oven glass than from any recipe. Oven too cold (learning rate too low): the loaf technically bakes but takes all day. Too hot: the crust burns while the inside stays raw — loss spikes and diverges. There's also a rhythm to good baking: start gentler while the dough sets (warmup), bake at full heat, then lower the temperature at the end so the inside finishes without burning (learning-rate decay). And the honest test is never the loaf's smell in the kitchen — it's giving a slice to someone who didn't watch it bake (validation).
Overfitting is the pastry chef who memorizes the exam: performance on practice questions keeps "improving" while real-world performance quietly rots. The fixes all share one idea — make memorizing harder than generalizing: shrink the dials toward zero unless the data insists (weight decay), randomly send some bakers home each shift so no single one becomes indispensable (dropout), or simply stop baking at the moment the outside taster was happiest (early stopping). Today you learn to read the oven glass — loss curves — like a clinician reads a chart: every squiggle has a differential diagnosis and a treatment.
"My model isn't learning" and "my fine-tune got worse" are the two most common deep-learning support tickets you will ever field, and both are answered by reading curves, not by re-running with hope. On Day 99 you will watch your GPT's train/val curves split and must decide: more data, more dropout, or stop? On Day 128, LoRA fine-tuning lives or dies by learning rate and early stopping. Being the person who looks at a loss chart and says "classic overfitting from step 3k, roll back and add weight decay" — that is billable judgment.
Reading loss curves like a clinician
step 1 / 5The healthy patient: train and validation loss fall together and flatten together. Small persistent gap is normal — the model has learned general patterns.
Guided practice
Manufacture all three pathologies on purpose
25 min- (Local run.) Reuse trainer.py and the arcs dataset from Day 88, but shrink the training set to 60 points to make overfitting easy, and train a 2-64-64-2 MLP.
- Baseline: lr=1e-2, 200 epochs. Plot (or print) train and val loss per epoch. You should see the val curve bottom out and creep up while train keeps falling — record the epoch where the gap opens.
- Underfit: swap in a 2-2-2 model with lr=1e-4. Both curves crawl and plateau high — note how different this signature looks.
- Explode: set lr=1.0 on the big model. Watch loss spike or NaN within a few epochs.
- Save the three histories to one JSON — they are your reference chart pack for Day 90 and Day 99 diagnosis.
Treat the overfitter three ways
20 min- Take the overfitting setup from exercise 1 and apply ONE treatment at a time, retraining from scratch each run (same seed): - weight decay: AdamW(..., weight_decay=0.01) - dropout: insert nn.Dropout(0.3) after each ReLU - early stopping: patience 15 on validation loss, restore the best checkpoint
- For each: record best validation loss and the epoch it occurred.
- Build a 4-row comparison table (baseline + three treatments) in your notes. Which single treatment helped most on THIS problem? Would you expect the same ranking with 100x more data? Why not?
- Combine the best two treatments and see if they stack.
Warmup and clipping rescue the hot oven
10 min- Return to the lr=1.0 exploding configuration — we will tame it without lowering the peak LR (a realistic exercise: big-model recipes often need high LRs to converge fast).
- Add linear warmup over the first 100 steps: each step, set
for g in opt.param_groups: g['lr'] = peak_lr * min(1.0, step / 100). - Add
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)between backward and step. - Rerun: training should now survive (even if final quality is mediocre — the point is stability). Note which intervention mattered more by toggling them individually.
On your own
The curve-reading clinic
20 minWrite diagnose(history) in dynamics.py: given {'train_loss': [...], 'val_loss': [...]}, return one of 'underfitting', 'overfitting', 'unstable', or 'healthy', with a one-line reason string. Rules are yours to design, but they must correctly classify: (a) your three pathology histories from guided 1, (b) the healthy Day-88 arcs run, and (c) a synthetic history you construct where val loss is flat-high while train loss is near zero.
Constraints: pure function, no plotting, no ML — just thresholds and slopes you justify in comments.
Hints: overfitting = val minimum well before the end plus a widening gap; instability = any NaN or a loss jump > 3x the running median; underfitting = both curves flat and high relative to their start.
The training playbook, page one
Commit dynamics.py (the diagnose function + tests against your saved histories) and playbook.md to your practice repo. The playbook is a one-page decision chart in markdown: four curve signatures, each with its diagnosis, first-line treatment, and second-line treatment, plus your starting recipe (optimizer, LR, warmup, clipping, when to add dropout vs weight decay) written as defaults you will actually reuse. Cap it at 40 lines — a playbook you can't skim during an incident is a diary. Day 90's ablation log and Day 99's GPT training log must both cite entries from this playbook.
Common mistakes & misconceptions
- Judging training by train loss alone. Train loss almost always falls; the val curve is where truth lives. No val curve = no diagnosis.
- Reaching for dropout the moment val loss rises, without checking data quantity first. With tiny data the honest fixes are more data or a smaller model; regularization is a treatment, not a cure.
- Using vanilla Adam with weight_decay and expecting L2 behavior — Adam couples decay with its adaptive scaling. Use AdamW; this distinction is why it exists.
- Forgetting that dropout and batch norm change behavior between train and eval modes — Day 88's eval() lesson is half of today's toolkit working correctly.
- Treating loss stuck at ln(num_classes) as "needs more epochs". That plateau is the uniform-guess score — suspect shuffled labels, a double softmax, or lr=0 before tuning anything.
- Cranking gradient clipping tighter and tighter to survive. Clipping is insurance for rare spikes; if every step clips, your LR or data pipeline is the real patient.
Q1. Train loss falls steadily; validation loss falls, bottoms out at epoch 30, then climbs. Diagnosis and first-line treatment?
Q2. A 10-class classifier's loss sits at ~2.30 for 20 epochs. The FIRST thing to check is…
Q3. Why does warmup help, especially for transformers?
Go deeper — curated resources
- Karpathy — A Recipe for Training Neural Networks — Search for this classic blog post: a battle-tested checklist (overfit one batch first, visualize everything, start simple) that professionalizes everything from today. Its "overfit a single batch" test is Day 97's sanity check.
- Three pathology runs produced, saved, and visually distinguished
- Treatment table filled with best-val-loss per remedy; stacking tried
- diagnose() passes on all five histories
- playbook.md committed with concrete default numbers
- Quiz ≥ 2/3
← Back: This is Day 76's bias/variance clinic re-run at neural scale: weight decay IS L2, early stopping is validation discipline, and the ln(10) plateau is Day 62's cross-entropy of a uniform guess.
Forward →: Tomorrow (Day 90) you run real ablations on MNIST using this playbook. Day 99's GPT training log is a curve-reading exam, and Day 128's LoRA runs make LR + early stopping the whole game. The loss-curve clinic returns as eval-metric drift on Day 145.
Unlocks: D90 MNIST Lab · D95 The Transformer