Evaluation Metrics
- Demonstrate the accuracy trap on imbalanced data and explain why it fools stakeholders
- Read a confusion matrix and compute precision, recall, and F1 from its four cells
- Choose between ROC-AUC and PR-AUC based on class balance and what the curve hides
- Pick regression metrics (MAE/RMSE/R²) and explain what each punishes
- Derive the right metric from a business harm model — cost of a miss vs cost of a false alarm
| Spaced-rep warm-up: Day 72 threshold cards + Day 59 base-rate card | 10 min |
| ELI5 + tech read; confusion-matrix visualizer | 20 min |
| Guided: name the counts + ROC vs PR | 38 min |
| Practice: metric memo clinic | 18 min |
| Project: metrics upgrade to ml_toolkit.py | 24 min |
| Quiz + flashcards | 10 min |
Builds on: Day 72 — Thresholds & predict_proba · Day 71 — Baselines & honest splits · Day 59 — Bayes and base rates
A security guard is graded on "percentage of correct decisions." The building gets broken into twice a year; thousands of innocent people walk in daily. The guard who waves EVERYONE through scores 99.9% — he was "right" about every innocent person and only wrong twice. Perfect grade, useless guard. The grade ignored the only events that mattered because they were rare.
Fair grading needs more than one number. Ask four questions instead: of the people he stopped, how many were actually burglars (precision — is he crying wolf)? Of the actual burglars, how many did he stop (recall — is he catching what matters)? What did each miss cost, and what did each false alarm cost? The right report card weights the questions by those costs. A cancer screener and a spam filter are both guards, but they must be graded oppositely: the screener is forgiven false alarms and fired for misses; the spam filter is forgiven misses and fired for false alarms. Today you learn to build the report card before the model — because the metric IS the definition of success.
"What metric?" is the first question in every ML interview and every customer engagement, and "accuracy" is the answer that fails both. FDEs inherit dashboards celebrating 97% accuracy on 3%-churn data — the model literally never predicts churn, and nobody noticed. Choosing the metric from the harm model is a conversation you lead with the customer, not a technicality: "what does a missed fraud cost you? a falsely-frozen account?" That conversation returns verbatim on Day 134 when you design LLM evals, and Day 83's churn project is graded on metric justification before model quality.
Grading fairly — one table, four kinds of right and wrong
step 1 / 6| pred fraud | pred legit | |
|---|---|---|
| actual fraud | TP ? | FN ? |
| actual legit | FP ? | TN ? |
A fraud model judged on 1,000 transactions. Rows = the truth, columns = the model's call. Every prediction lands in exactly one of four boxes.
Guided practice
Name the counts you tallied on Day 72
20 min- Paste the starter — the same 5%-positive imbalanced setup from Day 72, now graded properly. Standalone; run locally if the browser lacks scikit-learn.
- Run it. The dummy classifier posts ~95% accuracy with recall 0.00 — say the accuracy-trap sentence out loud: "it never once predicted the event we care about."
- Read the model's confusion matrix. Hand-compute precision and recall from the four cells with a calculator, then check them against classification_report. This closes the loop on Day 72's raw counts: caught = TP, missed = FN, false alarms = FP.
- Recompute precision/recall at thresholds 0.3 and 0.7 (reuse the proba array). Watch them trade. Write one sentence: which threshold would a fraud team pick, and which would a spam filter pick?
ROC-AUC flatters; PR-AUC tells
18 min- On the same imbalanced split, compute
roc_auc_scoreandaverage_precision_score(the PR-AUC estimate) for the logistic model. - Now make the problem rarer: regenerate with
weights=[0.99, 0.01]and retrain. Compare how much each metric moved. ROC-AUC will look almost unbothered; PR-AUC drops hard — precision is being destroyed by false positives among the flood of negatives, and only PR-AUC sees it. - Sanity anchor: what is a random model's ROC-AUC (0.5, always) and a random model's PR-AUC (the positive rate — 0.01 here)? Write both down; interviewers ask exactly this.
- Decision drill, one sentence each: which single curve-metric do you headline for (a) a 50/50 sentiment classifier, (b) a 1%-fraud detector? Why?
On your own
The metric memo clinic
18 minThree clients, three problems. For each, write a 3–4 sentence memo naming: the headline metric, the operating threshold philosophy (recall-first or precision-first), and one metric you refuse to report alone and why.
(a) Hospital sepsis early-warning: sepsis is ~2% of admissions; a miss can be fatal; a false alarm costs a nurse 10 minutes. (b) E-commerce fraud blocker: fraud ~0.5% of orders; a false block loses a real customer; a miss loses the goods. (c) House-price estimator for a bank: errors above 20% of the price are catastrophic; small errors are fine.
Hints: (a) and (b) are both rare-positive but their harm models point OPPOSITE directions on the precision/recall trade. (c) is regression — which of MAE/RMSE matches "large errors are catastrophic"?
Upgrade the toolkit to speak metrics
Refactor evaluate_classification in ml_toolkit.py: replace Day 72's raw caught/missed/false-alarm counts with a proper metrics dict — accuracy, precision, recall, F1, ROC-AUC, PR-AUC — for both model and dummy baseline, plus the confusion matrix at a caller-chosen threshold. Add metric_advice(positive_rate): a tiny function returning a one-line printed warning when the positive rate is under 15% ("imbalanced: headline PR-AUC, not accuracy/ROC-AUC"). Demonstrate in __main__ on the 5% dataset showing the dummy's 95% accuracy next to its 0.0 recall — the accuracy trap, permanently encoded in your own tooling. Commit.
Common mistakes & misconceptions
- Reporting accuracy on imbalanced data. A majority-class dummy scores (1 − positive rate) by doing nothing; always show the confusion matrix or precision/recall alongside.
- Using F1 as a default without asking whose harm model it encodes. F1 weights false positives and false negatives equally — most businesses do not.
- Headlining ROC-AUC for rare-positive problems. The false-positive RATE barely moves when negatives are plentiful; PR-AUC exposes the precision collapse ROC hides.
- Comparing precision or recall between models at different thresholds. Fix the threshold (or compare full curves); otherwise you are comparing policies, not models.
- Treating RMSE and MAE as interchangeable. RMSE squares errors, so one outlier can dominate it; MAE is the honest "typical miss." Choose by whether big errors are disproportionately bad.
- Trusting predicted probabilities from forests/boosting for expected-cost math without a calibration check. Ranking can be great while the probabilities are fiction.
Q1. A churn model on 4%-churn data reports 96% accuracy. What must you check before celebrating?
Q2. Your fraud model flags 200 transactions; 40 are real fraud. There were 100 frauds in total. Precision and recall are…
Q3. For a 1%-positive problem, why prefer PR-AUC over ROC-AUC as the headline?
Go deeper — curated resources
- Cost-sensitive thresholding — With calibrated probabilities and per-cell dollar costs, the optimal threshold is where expected cost of flagging equals expected cost of not flagging — turning Day 72's sweep into one formula. Try deriving it for practice scenario (b).
- Precision and recall hand-computed from the confusion matrix and verified against sklearn
- ROC-vs-PR experiment run at 5% and 1% positives; floors for a random model written down
- Three metric memos written with harm-model justification
- ml_toolkit.py metrics upgrade committed
- Quiz ≥ 2/3
← Back: Day 72's threshold sweep tallied these exact cells before they had names; Day 59's base-rate neglect is the accuracy trap wearing a lab coat. Day 71's baseline discipline is why every metric here is reported against a dummy.
Forward →: Tomorrow cross-validation puts error bars on these metrics. Day 77's model card and Day 83's churn project both open with metric choice. On Day 134–139 the same discipline — metric from harm model, baseline first, error bars — becomes LLM eval design.
Unlocks: D77 Week 11 Checkpoint: Tabular Mini-Competition · D80 Error Analysis · D90 MNIST Lab · D134 Eval Mindset & Golden Sets