Day 81 · The lab notebook

Experiment Tracking & Reproducibility

You will be able to

Log params, metrics, and artifacts to MLflow so every run is findable and comparable
Name and organize experiments so future-you can answer "which run was that?" in seconds
Control randomness with seeds and state honestly what seeds do NOT guarantee
Pin the environment and data identity a result depends on
Compare runs honestly — same data, same split, same metric — before believing a delta

Today's ~120 minutes

Spaced-rep warm-up: Days 76–80 due cards	10 min
ELI5 + tech read	18 min
Guided: tracked bake-off + artifact bundle	37 min
Practice: the archaeology test	18 min
Project: tracked_bakeoff + conventions doc	25 min
Quiz + flashcards	12 min

Builds on: Day 76 — CV & honest comparison · Day 17 — Environments & pinning · Day 8 — Git — versioning code

The analogy

🧸 ELI5⚙ Tech

A chemist runs an experiment and scribbles the result on a napkin: "worked better!" Better than WHAT? At what temperature? Which batch of reagent? Six weeks later the napkin says nothing, the result cannot be reproduced, and the discovery is functionally lost. This is why real labs enforce the lab notebook: every run gets an entry — conditions, materials, measurements — before the scientist is allowed to have an opinion about the result.

Your ML work this week has been generating napkins. "The forest got 0.87" — with which features? which split seed? which hyperparameters? which version of the cleaning script? By Day 83 you will have dozens of runs, and by the capstone, hundreds; memory will not hold them and terminal scrollback is where results go to die. An experiment tracker is the enforced lab notebook: every run automatically records its parameters, scores, and outputs into a searchable table, so "which run was best, and what exactly produced it?" is a query, not an archaeology dig. Reproducibility is the notebook's other half: seeds, pinned packages, and versioned data — the "which batch of reagent" details that decide whether a rerun gets the same answer.

Why this matters on the job

"How do you track experiments?" is a standard ML-engineering interview probe, and "spreadsheet, mostly" is a red flag. In team settings the tracker IS the shared memory — an FDE who can pull up the exact run behind the number a customer is questioning, with its params and artifacts, earns trust that "I think it was around 0.87?" destroys. Reproducibility is also a contractual matter: regulated customers ask you to re-produce the shipped model from scratch. And the habit transfers directly: on Day 141 prompts and eval scores go through the same run-tracking discipline — different artifacts, identical hygiene.

Guided practice

guided 1

First tracked experiment: the bake-off, remembered forever

22 min

Locally: pip install mlflow (this lab needs a real terminal, not the browser interpreter). Paste the starter — it re-runs a compact version of Day 74's bake-off, but every run is logged.
Run it, then launch mlflow ui in the same directory and open http://127.0.0.1:5000. Find your experiment; sort runs by test_accuracy.
In the UI, select the forest and boosting runs and click Compare. The params table shows exactly what differed — this view is the whole point: no memory, no napkins.
Re-run the script with SPLIT_SEED = 1 at the top. Watch four new runs appear. In the UI, filter by the seed param and compare accuracy across seeds — Day 76's split-variance lesson, now permanently on record.
Note what got logged WITHOUT you thinking: start time, duration, source file. And note what did NOT: the data identity. The starter logs rows + a hash — confirm you can see them.

🐍 python — editable, runs in your browser

import hashlib
import mlflow
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

SPLIT_SEED = 0
X, y = load_breast_cancer(return_X_y=True)
data_hash = hashlib.md5(X.tobytes()).hexdigest()[:10]
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.25, random_state=SPLIT_SEED, stratify=y)

mlflow.set_experiment("day81-bakeoff")
models = [
    ("dummy", DummyClassifier(strategy="most_frequent"), {}),
    ("logreg", make_pipeline(StandardScaler(),
                             LogisticRegression(max_iter=2000)), {"C": 1.0}),
    ("forest", RandomForestClassifier(n_estimators=200, random_state=0),
     {"n_estimators": 200}),
    ("gbm", HistGradientBoostingClassifier(learning_rate=0.1, random_state=0),
     {"learning_rate": 0.1}),
]
for name, model, hp in models:
    with mlflow.start_run(run_name=f"{name}-seed{SPLIT_SEED}"):
        mlflow.log_params({"model": name, "split_seed": SPLIT_SEED,
                           "data_rows": len(X), "data_hash": data_hash, **hp})
        model.fit(X_tr, y_tr)
        mlflow.log_metric("train_accuracy", model.score(X_tr, y_tr))
        mlflow.log_metric("test_accuracy", model.score(X_te, y_te))
print("done — now run: mlflow ui")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Artifacts and the reproducibility bundle

15 min

Extend one run (the gbm) to log artifacts: write the classification report to report.txt and log it; pip freeze into requirements_frozen.txt (via subprocess or manually) and log that too; log the fitted model with mlflow.sklearn.log_model(model, name="model").
Add the git identity: git rev-parse HEAD as a tag (mlflow.set_tag("git_commit", ...)). If the working tree is dirty, tag dirty=true — a result from uncommitted code is only half-pinned.
In the UI, open the run and browse its artifacts. Everything needed to re-create or ship this exact model now lives in one place.
The seed honesty check: run the gbm five times with model seeds 0–4 (same split), logging each. Pull the five test accuracies from the UI and write the spread as "0.xx ± 0.0y" — that tolerance is what "reproducible" honestly means.

🐍 python — editable, runs in your browser

import subprocess
import mlflow
import mlflow.sklearn
from sklearn.metrics import classification_report

with mlflow.start_run(run_name="gbm-full-bundle"):
    mlflow.log_params({"model": "gbm", "learning_rate": 0.1,
                       "split_seed": SPLIT_SEED, "data_hash": data_hash})
    model = HistGradientBoostingClassifier(learning_rate=0.1, random_state=0)
    model.fit(X_tr, y_tr)
    mlflow.log_metric("test_accuracy", model.score(X_te, y_te))

with open("report.txt", "w") as f:
        f.write(classification_report(y_te, model.predict(X_te)))
    mlflow.log_artifact("report.txt")

frozen = subprocess.run(["pip", "freeze"], capture_output=True, text=True)
    with open("requirements_frozen.txt", "w") as f:
        f.write(frozen.stdout)
    mlflow.log_artifact("requirements_frozen.txt")

commit = subprocess.run(["git", "rev-parse", "HEAD"],
                            capture_output=True, text=True).stdout.strip()
    mlflow.set_tag("git_commit", commit or "no-git")
    mlflow.sklearn.log_model(model, name="model")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

The archaeology test

18 min

Simulate future-you with someone else's napkins. Without opening the scripts again, answer FROM THE MLFLOW UI ALONE: (1) which single run had the best test accuracy overall, and under exactly which params? (2) how much did split_seed 0 → 1 move the forest's accuracy? (3) what package version of scikit-learn produced the gbm-full-bundle run? (4) could you re-create the gbm-full-bundle model on a fresh machine — list what you would download from the run and what is still missing, if anything.

Goal: all four answered from the UI in under 10 minutes. Anything you could NOT answer is a logging gap — go fix the logging code so the next run captures it, and write one sentence on the gap.

Hints: (3) lives in the requirements artifact. For (4), the honest inventory is: model artifact + frozen env + git commit + data hash — the data itself is referenced, not stored; is that acceptable here? When would it not be?

Ship before you stop

Tracking enters the toolkit — and Day 83 inherits it

Add tracked_bakeoff(models, X_train, X_test, y_train, y_test, experiment, data_note) to ml_toolkit.py: your Day 74 bakeoff, with every model logged to MLflow (params, train/test metrics, gap) plus a shared data_note param (rows + hash) and git-commit tag per run. Make MLflow optional-but-loud: if import fails, run untracked and print a warning. Write tracking_conventions.md: your experiment-naming scheme, run-naming scheme, the four identities checklist (code/data/env/seed), and the five-seed tolerance protocol. Day 83's churn project REQUIRES tracked runs — this function is how. Commit both.

Rubric — check what you completed (0/5)

tracked_bakeoff logs params, both metrics, and the gap for every model in one experimentData identity (rows + hash) and git commit recorded on every runGraceful degradation when mlflow is not installed, with a visible warningConventions doc covers naming, the four identities, and the seed-tolerance protocolCommitted to the practice repo

Common mistakes & misconceptions

Logging only the runs that worked. Survivor-only notebooks create "it worked in March" myths; start the run record BEFORE fitting, so failures are archived too.
Comparing runs that differ in more than the thing being tested. If split seed AND features AND model changed, the delta is unattributable — the tracker shows this if you look at the params diff.
Believing a seed makes results reproducible everywhere. Seeds pin one environment; library versions, OSes, and thread schedules can still shift numbers. Claim "reproducible within measured tolerance."
Tracking metrics but not data identity. The silent killer is the CSV that changed under you; log rows + hash (or a real data version) so change is detectable.
Putting the tracking calls in a notebook you never re-run. Tracking belongs in the training SCRIPT (Day 82's refactor), so every execution is a record, not a ceremony.
Treating uncommitted code as loggable. A git hash with a dirty tree points at code that no longer exists; commit first or tag the run dirty so future-you knows to distrust it.

Knowledge check

Q1. A teammate asks "what produced the 0.87 from last Tuesday?" With proper tracking, the answer comes from…

Q2. You set random_state=42 everywhere. A colleague on a newer scikit-learn gets slightly different numbers. This means…

Q3. Which comparison between two tracked runs is actually valid?

Go deeper — curated resources

docsMLflow Documentation — tracking quickstart & concepts ↗30 min courseMade With ML — experiment tracking lesson ↗25 min repoDesigning ML Systems book repo (Chip Huyen) — reproducibility notes ↗15 min

If you have a third hour

Data versioning beyond a hash — A content hash detects change but cannot restore the old data. DVC-style tools version datasets like git versions code — worth a skim now; the capstone's ingest pipeline (Day 119+) faces exactly this problem for document corpora.

Done means

MLflow UI running locally with ≥ 9 logged runs across two seeds
Full-bundle run contains report, frozen requirements, git tag, and logged model
Archaeology test: 4/4 answered from the UI, gaps fixed in logging code
tracked_bakeoff + tracking_conventions.md committed
Quiz ≥ 2/3

How this connects

← Back: Day 76 taught you that single numbers lie and comparisons need identical protocols — the tracker is that discipline with a database. The environment pinning is Day 17's lockfiles, and the git hash per run is Day 8's time machine cross-referenced with results.

Forward →: Day 82 moves tracking into the reusable training script, and Day 83's churn rubric requires tracked runs. On Day 141 the same notebook records prompt versions and eval scores; the four-identities checklist reappears verbatim for LLM systems.

Unlocks: D82 ML Code Structure & Pipelines · D83 Phase Project: Churn Prediction End-to-End