Day 76 · Practice tests vs the real exam

Validation, Bias/Variance & Regularization

You will be able to

Run k-fold cross-validation and report mean ± std instead of a single lucky split
Diagnose bias vs variance from train/validation curves and prescribe the right fix
Explain what L1 and L2 penalties do to weights and when each is the right choice
Tune hyperparameters with GridSearchCV/RandomizedSearchCV without touching the test set
Describe the overfitting-to-the-validation-set trap and the nested-CV idea that guards against it

Today's ~120 minutes

Spaced-rep warm-up: Days 73–75 cards + Day 60 error-bar card	10 min
ELI5 + tech read; bias-variance visualizer	20 min
Guided: folds & error bars + honest tuning & L1	38 min
Practice: the diagnosis clinic	20 min
Project: cv_report, tune, and bakeoff upgrade	22 min
Quiz + flashcards	10 min

Builds on: Day 71 — Train/test splits & baselines · Day 74 — Ensembles & the bake-off · Day 60 — Sampling variance & confidence intervals

The analogy

🧸 ELI5⚙ Tech

A student has one practice test and one real exam. She takes the practice test, studies, retakes it, studies again — until she scores 100%. Has she learned chemistry, or has she learned THAT practice test? You cannot tell, and neither can she. The honest setup: five practice tests, rotated — study on four, check yourself on the held-out fifth, then rotate which one is held out. Five scores instead of one, and their average AND their spread tell you how she will really do. The real exam stays sealed in an envelope until the very end, taken exactly once.

That is cross-validation. The rotation is k-fold; the sealed envelope is your test set. And the student's failure mode has a name in ML: tune your model against the same validation data enough times and you overfit to the validation set itself — the practice test memorized by proxy. The spread across folds is the error bar Day 60 taught you to demand: a model that scores 0.85 ± 0.01 across folds and one that scores 0.86 ± 0.06 are not simply "0.86 wins."

Why this matters on the job

Every result you report from now on carries this discipline: mean ± spread across folds, test set touched once. Interviews probe it directly ("your val score improved — how do you know it's real?") and bias/variance diagnosis is a top-three ML interview topic. In the field it is the difference between "our model got better" and shipping a mirage: FDEs see customers burn weeks tuning against one lucky split. Day 63 already made you demand error bars on eval scores; Day 139 applies the identical logic to LLM evals, where run-to-run variance is even nastier.

Watch it happen

Practice tests vs the real exam — error as complexity grows

step 1 / 5

train error

Train a family of models, simple to complex, and track TRAINING error. It only goes down: a bigger model can always memorize more of the practice test.

🎯 Predict modespeed1×

Guided practice

guided 1

One split lies; five folds put error bars on it

20 min

Paste the starter. It scores logistic regression on the breast-cancer data with ten different single train/test splits, then with 5-fold CV. Standalone; run locally if needed.
Run it. Look at the ten single-split scores: the spread is your Day 71 "change random_state and the score moves" observation, quantified. A single split is one draw from that distribution.
Compare with the CV result: mean ± std from five folds. Write the reporting sentence you will use forever: "accuracy 0.97 ± 0.01 across 5 folds."
Now the leakage drill: the starter scores TWO pipelines — scaler inside the pipeline (correct) vs scaling X once before CV (subtly wrong: each fold's scaler saw the validation rows). The scores differ little here, but write one sentence on why the second is disqualified on principle — the fold was not truly held out.

🐍 python — editable, runs in your browser

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

singles = []
for seed in range(10):
    X_tr, X_te, y_tr, y_te = train_test_split(
        X, y, test_size=0.25, random_state=seed, stratify=y
    )
    clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
    singles.append(clf.fit(X_tr, y_tr).score(X_te, y_te))
print("ten single splits:", [round(s, 3) for s in singles])
print(f"spread: min={min(singles):.3f} max={max(singles):.3f}")

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
scores = cross_val_score(pipe, X, y, cv=5)
print(f"5-fold CV: {scores.mean():.3f} +/- {scores.std():.3f}")

# the disqualified version: scaler fit on ALL rows before CV
X_leaky = StandardScaler().fit_transform(X)
leaky = cross_val_score(LogisticRegression(max_iter=2000), X_leaky, y, cv=5)
print(f"leaky-scaled CV: {leaky.mean():.3f} +/- {leaky.std():.3f}  (do not do this)")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Tune without cheating, then watch L1 delete features

18 min

Split breast-cancer into train/test ONCE (the sealed envelope). On the training part only, run GridSearchCV over logistic regression's C in [0.01, 0.1, 1, 10, 100] with 5-fold CV inside a scaling pipeline.
Print search.best_params_ and search.best_score_ (the CV estimate), then — once — score search.best_estimator_ on the sealed test set. The two numbers should be close; a big drop means the search overfit the folds.
Now the L1 demo: fit LogisticRegression with penalty="l1", solver="liblinear" at C = 1, 0.1, 0.05 and count nonzero coefficients each time (30 features total). Watch features get zeroed as the penalty tightens — automatic feature selection.
Write the pairing in your notes: L2 = shrink everything, keep everything (stability); L1 = shrink and DELETE (sparsity/selection). Both need scaled features.

🐍 python — editable, runs in your browser

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

pipe = Pipeline([("scale", StandardScaler()),
                 ("clf", LogisticRegression(max_iter=5000))])
search = GridSearchCV(pipe, {"clf__C": [0.01, 0.1, 1, 10, 100]}, cv=5)
search.fit(X_tr, y_tr)
print("best C:", search.best_params_["clf__C"],
      " CV score:", round(search.best_score_, 3))
print("sealed test score:", round(search.score(X_te, y_te), 3))

scaler = StandardScaler().fit(X_tr)
for C in [1, 0.1, 0.05]:
    l1 = LogisticRegression(penalty="l1", solver="liblinear", C=C)
    l1.fit(scaler.transform(X_tr), y_tr)
    nz = int(np.sum(l1.coef_ != 0))
    print(f"L1 C={C:5.2f}  nonzero coefficients: {nz}/30")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

The diagnosis clinic

20 min

Generate a small noisy dataset: make_classification(n_samples=400, n_features=20, n_informative=5, flip_y=0.05, random_state=3). Produce learning curves (sklearn.model_selection.learning_curve, train sizes 50→300, cv=5) for two patients: (a) LogisticRegression in a scaled pipeline, (b) an unconstrained DecisionTreeClassifier.

Goal: (1) tabulate train vs validation score at each size for both; (2) diagnose each patient in bias/variance language, citing the gap and the trend; (3) prescribe: for each, would more data help, and what single change would you make instead or as well? (4) verify one prescription by applying it and re-running.

Hints: the tree should show a huge persistent gap (variance — cap depth or add data); logistic on 20 mostly-noise features may show both curves converging early (bias-ish for the nonlinear part it cannot express — but also note how L1 could prune the 15 noise features).

Ship before you stop

CV and search enter the toolkit

Add two functions to ml_toolkit.py. cv_report(model, X, y, cv=5, scoring=None) returns mean, std, and the per-fold scores, and formats the "0.97 ± 0.01" reporting string. tune(model, param_grid, X_train, y_train, cv=5) wraps GridSearchCV, prints best params + CV score, and returns the refit best estimator — with a docstring warning that the returned CV score is optimistic (best-of-many) and the sealed test set is scored once, elsewhere. Upgrade bakeoff from Day 74 to accept cv=k and report mean ± std per model instead of single-split scores. Demonstrate on breast-cancer in __main__. Commit.

Rubric — check what you completed (0/5)

cv_report returns mean/std/fold-scores and the formatted ± stringtune performs search on training data only and documents the optimism caveat in its docstringbakeoff(cv=5) reports mean ± std per model and still sorts sensiblyPipelines used so scaling happens inside folds — no leaky preprocessingCommitted to the practice repo

Common mistakes & misconceptions

Reporting the best GridSearchCV score as your generalization estimate. It is the max of noisy numbers — optimistically biased. The sealed test set (or nested CV) gives the honest figure.
Running CV on pre-scaled data. Fitting the scaler on all rows lets every fold peek at its validation part. Put preprocessing in the Pipeline; CV the pipeline.
Tuning against the test set "just once or twice." Each peek converts exam into homework; by the fifth peek your test score is a validation score with a fancy name.
Prescribing more data for a high-bias model. If train and validation scores are low and converged, the model cannot express the signal — capacity or features, not rows.
Using L1 on unscaled features and reading the surviving coefficients as importance. The penalty hits large-unit features hardest; scale first or the selection is an artifact of units.
Ignoring the std across folds. A 0.86 ± 0.06 model is not better than 0.85 ± 0.01 — Day 60's error-bar discipline applies to model selection too.

Knowledge check

Q1. Model A: train 0.97, 5-fold validation 0.71. Model B: train 0.74, validation 0.72. Diagnoses?

Q2. Why must the scaler be fit inside each CV fold (via a Pipeline) rather than once on all the data?

Q3. L1 vs L2 regularization — the practical difference?

Go deeper — curated resources

docsscikit-learn User Guide — 3.1 Cross-validation ↗30 min videoStatQuest — bias & variance, ridge/lasso regression ↗25 min courseGoogle ML Crash Course — generalization & regularization ↗20 min courseKaggle Learn — Intermediate ML (cross-validation lesson) ↗20 min

If you have a third hour

Nested cross-validation, properly — Outer CV estimates the generalization of the WHOLE procedure (search included); inner CV does the tuning. Expensive but the honest answer when no test set can be spared — sklearn's docs show it in ~15 lines.

Done means

Ten-split spread vs CV mean ± std recorded; reporting sentence written
GridSearchCV run with sealed-test confirmation; L1 sparsity counts recorded
Both learning-curve patients diagnosed and one prescription verified
Toolkit functions committed
Quiz ≥ 2/3

How this connects

← Back: Day 60's sampling variance is why one split lies, and Day 61's multiple-comparisons warning is why the best-of-grid score flatters. Day 73's depth sweep was bias/variance before it had the name; Day 72's C parameter finally got its full story.

Forward →: Tomorrow's competition requires CV and a sealed holdout — this is the referee's rulebook. Day 83's churn rubric grades CV discipline explicitly. On Day 139, fold-to-fold variance becomes run-to-run variance in LLM evals, same statistics, higher stakes.

Unlocks: D77 Week 11 Checkpoint: Tabular Mini-Competition · D81 Experiment Tracking & Reproducibility · D88 Training Loops & Data · D89 Training Dynamics