Week 11 Checkpoint: Tabular Mini-Competition
- Recall the week's core contrasts from memory: under/overfitting, bagging/boosting, precision/recall, L1/L2
- Beat two baselines on a held-out split of a real tabular dataset using the full Week 11 protocol
- Write a model card: metric choice, CV setup, results with error bars, error analysis, next steps
- Apply the sealed-test-set discipline under mild competitive pressure without cheating
| Recall drills: closed-book Week 11 spine, then diff | 25 min |
| Competition setup + metric declaration | 10 min |
| The competition: baselines → model → CV → seal-break | 45 min |
| Model card writing + error autopsy | 25 min |
| Cumulative quiz + flashcard deck review (due cards D71–76) | 15 min |
Builds on: Day 71 — Baselines & framing · Day 74 — Ensemble bake-off · Day 75 — Metric choice · Day 76 — CV & honest tuning
A week of driving lessons is not a driver. Today you take the road test: one real dataset, a locked-away holdout, and a leaderboard of exactly three entries — a dummy baseline, a plain logistic regression, and you. Your only job is to beat the first two, honestly, and to write up how you did it.
Why a competition format? Because mild pressure is a truth serum. When a score is at stake, the temptation to peek at the test set, to tune one more time against the same folds, or to quietly switch metrics until one looks good becomes real — and resisting those temptations is the actual skill being tested. And why start with recall drills before touching the data? Spaced repetition: six days of concepts are just now starting to fade, and the act of dragging them back from memory — not rereading them — is what moves them to long-term storage. You will need them fluent, not familiar: the ML interview on Day 84 and every real project after it assume this week is reflex.
This is the smallest complete rehearsal of the job. Real engagements are exactly this shape: unfamiliar tabular data, a business metric to justify, baselines to beat, and a writeup a stakeholder can trust. The model card you produce today is a portfolio artifact — hiring managers ask "show me an end-to-end" and this is your first. It is also the dress rehearsal for Day 83's graded churn project: same protocol, higher stakes. Kaggle-style competition experience, even a private one, is a recognized signal that you can operationalize theory.
Guided practice
Recall drills — closed book, then diff
25 minClose all notes. In a fresh markdown file, from memory:
- Draw the 2×2 confusion matrix, label all four cells, and write the precision and recall formulas. (Day 75)
- Write the bias/variance diagnosis table: the two score signatures and two fixes for each. (Day 76)
- Explain bagging vs boosting in exactly two sentences. (Day 74)
- Write the log-loss formula and say in one line why MSE is wrong for classification. (Day 72)
- List the three-choice framing of any ML setup and the first model you fit on any new problem. (Day 71)
- Now open the week's notes and diff. Mark each item ✓ (clean), ~ (partial), ✗ (blank). Re-derive every ✗ from the source day before proceeding — the afternoon build depends on them.
Competition setup — the referee's checklist
10 min- Create
week11_competition/in your practice repo with the starter script. - Run the data-loading cell once; confirm shapes (~48k × 14) and the positive rate (~0.24).
- Write your metric declaration at the top of the model card BEFORE fitting anything: headline metric, why, and the threshold philosophy if applicable.
- Verify the sealed split exists and is stratified. From this point, X_test/y_test appear in exactly one more line of code — the final scoring.
On your own
The competition itself
45 minBeat the two baselines on the sealed test set. Constraints: categorical features need encoding (Day 69's ColumnTransformer with OneHotEncoder(handle_unknown="ignore") for linear models; OrdinalEncoder is fine for tree ensembles), all model selection happens via CV on the training portion, and the test set is scored once, at the end, for all three entries together.
Suggested path (not mandatory): baselines → HistGradientBoostingClassifier default → one tuning pass (learning_rate × max_iter via your toolkit's tune()) → final CV check → seal-break.
Hints if stuck: HistGradientBoosting accepts categorical features natively via categorical_features="from_dtype" on pandas categoricals — one line instead of an encoder. A good boosted model lands around 0.87 accuracy / 0.92 ROC-AUC territory; your leaderboard should show clear water between all three entries. If your model barely beats logistic regression, say so honestly in the card — that finding is common on this dataset and worth two sentences.
Ship the model card
Write week11_competition/MODEL_CARD.md: (1) framing — target, features, who would use this prediction; (2) metric declaration written before modeling, with the harm-model sentence; (3) data notes — at least two observations (imbalance, a suspicious feature, missing values); (4) method — encoding choice, CV setup, what you tuned; (5) the three-entry leaderboard table from the single seal-break, with CV mean ± std alongside; (6) error autopsy — three test-set misclassifications examined, one hypothesis each; (7) next steps — two concrete ideas you did NOT do. Commit the card and the runnable script together.
Common mistakes & misconceptions
- Tuning "one more time" after seeing the test score. The seal is broken — any further changes are test-set fitting. The honest move: report it, and note the re-tune as future work.
- Choosing the metric after seeing which one your model wins. Metric declaration precedes modeling; that ordering is the integrity of the whole exercise.
- One-hot encoding into a tree ensemble and wondering why it slowed down. Trees handle ordinal/native categoricals well; one-hot explodes width for no gain there (it IS needed for linear models).
- Reporting only the test number. Test-once gives a point estimate; the CV mean ± std is what makes it credible. Report both.
- Skipping the error autopsy because the score looked good. Three examined errors routinely reveal a data bug or a slice problem the aggregate metric hides — Day 80 builds a whole day on this.
- Treating the recall drills as optional warm-up. Retrieval practice is the highest-leverage 25 minutes of the day; rereading feels better and teaches less.
Q1. On this ~24%-positive dataset, your model posts accuracy 0.87. What single number do you check before trusting it?
Q2. Your GridSearchCV best score is 0.90 but the sealed test gives 0.87. The MOST likely explanation is…
Q3. You one-hot encode 14 mixed columns and feed them to logistic regression AND to gradient boosting. Which pairing is questionable?
Go deeper — curated resources
- Model Cards for Model Reporting (Mitchell et al.) ↗ — The paper that named the artifact you wrote today — skim the proposed sections and note which ones your card already covers.
- All six recall drills attempted closed-book; every ✗ re-derived
- Leaderboard shows your model beating both baselines on the once-touched test set
- MODEL_CARD.md complete with all seven sections and committed
- Cumulative quiz ≥ 2/3
← Back: Every rule in today's protocol was earned this week: baselines (Day 71), threshold thinking (Day 72), memorization risk (Day 73), the ensemble lineup (Day 74), metric-from-harm (Day 75), CV and the sealed envelope (Day 76). The ColumnTransformer encoding is Day 69 paying rent.
Forward →: Day 83 reruns this protocol on churn with an 8-criterion rubric and a business framing layer. The error autopsy expands into Day 80's full error-analysis method, and the model card format returns for LLM systems on Day 140's eval report.
Unlocks: D80 Error Analysis · D83 Phase Project: Churn Prediction End-to-End · D84 Week 12 Checkpoint + Interview Drill II