Error Analysis
- Slice evaluation metrics by subgroup and find where a "good" aggregate hides a bad pocket
- Drive data and feature fixes from confusion patterns instead of blind model swaps
- Distinguish data-centric from model-centric iteration and pick the higher-leverage move
- Spot probable label noise by inspecting high-confidence errors
- Run a feature ablation and read what actually carries the signal
| Spaced-rep warm-up: Days 75–79 due cards | 10 min |
| ELI5 + tech read | 18 min |
| Guided: find the sick slice + error ledger | 38 min |
| Practice: feature ablation | 18 min |
| Project: playbook + toolkit functions | 26 min |
| Quiz + flashcards | 10 min |
Builds on: Day 75 — Confusion matrix & sliced metrics · Day 77 — The competition's error autopsy · Day 67 — EDA instincts
A patient says "I feel bad." A poor doctor prescribes something general and hopes. A good doctor reads the chart: WHEN is it bad — mornings? after meals? The pattern is the diagnosis: "after meals" points to the stomach, "mornings" points elsewhere. Same complaint, different disease, different treatment — and the treatment chosen without the pattern is a guess with a prescription pad.
"My model is 86% accurate" is the patient saying "I feel bad." Error analysis is reading the chart: WHERE do the errors live? Split them by every dimension you have — customer type, value range, class — and the aggregate almost always shatters into unequal pieces: 95% here, 61% in that pocket over there. Now you have a diagnosis, and the treatment writes itself: that pocket needs more data, or a feature the model lacks, or its labels are simply wrong. The discipline is embarrassingly simple — LOOK at your errors, one by one, and count what you see — and it is the single highest-leverage habit in applied ML, mostly because almost nobody does it.
Andrew Ng's data-centric argument matches what FDEs live daily: after the first decent model, the metric usually moves further from fixing data than from swapping architectures — and error analysis is what tells you WHICH data. The skill transfers wholesale to LLM work: on Days 136 and 143 you will read failed RAG answers and cluster production failures, which is today's method with transcripts instead of rows. And it is a differentiator in interviews: "my model underperforms — walk me through your next hour" separates candidates who tune blindly from engineers who diagnose. Your Day 83 churn rubric contains an error-analysis criterion for exactly this reason.
Guided practice
Find the sick slice
22 min- Paste the starter. It builds a synthetic loans dataset with a planted disease: for one customer segment ("gig" workers), the informative income feature is replaced by noise — the model will do fine everywhere and badly there. It also flips 3% of labels at random (noise you will hunt in the next exercise).
- Run it. The aggregate accuracy looks healthy. Now read the per-segment table: find the segment scoring far below the rest, and note its share of the test set.
- For the sick segment, print the confusion counts. Which error type dominates? Write the one-line diagnosis: what does the model not know for these rows, and what would you ask the customer for?
- Confirm the diagnosis with the counts caveat: the segment has hundreds of rows here, so it is signal. Recompute the aggregate accuracy EXCLUDING the sick segment to see what the model could be if the data gap were fixed.
The error ledger + the label-noise sniff
16 min- Pull the 20 test errors where the model was MOST confident: sort errors by
abs(proba - 0.5)descending and take the top 20. - For each, print the row: segment, features, true label, predicted probability. Ledger each with a one-word hypothesis: "gig-gap" (sick slice), "label?" (features scream one class, official label says the other), or "hard" (genuinely ambiguous).
- Count your ledger. Because the generator planted both diseases, your counts have an answer key: the starter prints how many of your top-20 confident errors carry the flip flag. Check your "label?" hypotheses against it — high-confidence errors should be heavily enriched for flipped labels versus the 3% base rate.
- Write the takeaway sentence: aggregate metrics said "fine"; thirty minutes of reading errors produced two concrete fixes (get real gig income data; audit labels) — neither of which is "try a bigger model."
On your own
Ablation: what carries the signal?
18 minOn the loans dataset (before dummies), run a feature ablation with 5-fold CV: full feature set, then drop one of [income, debt, tenure, segment] at a time, retrain, record the CV accuracy delta from full.
Goal: (1) produce the ablation table sorted by damage; (2) reconcile it with the generator's ground truth (which coefficients were large?); (3) explain the segment column's delta: it carries little DIRECT signal, so why might dropping it still hurt slightly? (4) one sentence on why ablation beats feature_importances_ for this question.
Hints: use your toolkit's cv_report. The segment dummies let the model partially route around the gig income gap — an interaction effect importances tend to smear. Redundant-feature caveat: income and debt are independent here, but in real data correlated pairs shield each other.
The error-analysis playbook, executable
Add slice_report(df, slice_cols, y_col="y", pred_col="pred") to ml_toolkit.py: for each named column it returns per-value metric, count, and delta-from-aggregate, flagging slices that are both below aggregate AND large enough to matter (count ≥ 30). Add confident_errors(df, k=20) returning the top-k most confident mistakes for ledger reading. Then write error_analysis_playbook.md — your personal checklist, ≤ 1 page: the four steps (slice → ledger → count hypotheses → pick data-vs-model move), the counts caveat, the label-noise sniff, the ablation recipe. You will follow this playbook verbatim on Day 83 and again on Day 136 for RAG. Commit both.
Common mistakes & misconceptions
- Iterating on the aggregate metric alone. A model can improve overall while a critical slice worsens; slicing is how you notice before the customer does.
- Trusting tiny slices. A 3/5 accuracy slice is a coin flip with a story. Report counts with every slice metric and set a minimum-n before reacting (Day 60's error bars).
- Blaming the model for label noise. When a strong model is confidently "wrong," audit the label first — high-confidence errors are enriched for annotation mistakes.
- Reaching for a bigger model as the first response to errors. After a strong baseline, the ledger usually points at data: a missing feature, a broken slice, bad labels. Diagnose, then treat.
- Reading a zero-delta ablation as "useless feature." Correlated features cover for each other; the honest claim is "redundant given the rest," which is different from "uninformative."
- Doing error analysis once. It is a loop: fix, retrain, re-slice — fixes shift where the errors live, and yesterday's healthy slice can be today's sick one.
Q1. Overall accuracy 88%; the "enterprise" segment (25% of traffic) scores 64%. Best next move?
Q2. Your model assigns p = 0.98 to a class and the label disagrees. This pattern, repeated, is most often a sign of…
Q3. You drop a feature and CV accuracy doesn't move. The correct conclusion is…
Go deeper — curated resources
- Confident learning & the cleanlab idea — The label-noise sniff, industrialized: use out-of-fold predicted probabilities to estimate which labels are likely wrong across the whole dataset. Worth a skim of the cleanlab README after today's hand-rolled version.
- Sick segment found, diagnosed, and the excluding-segment ceiling computed
- Error ledger for 20 confident errors completed; label-noise enrichment verified against the answer key
- Ablation table produced and reconciled with ground truth
- slice_report, confident_errors, and playbook committed
- Quiz ≥ 2/3
← Back: Day 77's three-error autopsy was the trailer; today was the method. The slicing is Day 66's groupby pointed at mistakes, the confusion patterns are Day 75's cells per slice, and the counts caveat is Day 60 refusing to let five rows tell a story.
Forward →: Day 83's churn rubric grades this playbook in action. On Day 136 the ledger reads failed RAG answers, on Day 143 you cluster production failures at scale (Day 78 + today), and Day 90's MNIST error analysis applies it to a neural net's confusion matrix.