Week 9 Checkpoint: Math Assessment
- Reproduce the week's core formulas (Bayes, standard error, CI, entropy, cross-entropy) from memory
- Score ≥ 16/20 on the cumulative math assessment and map every miss to a revisit day
- Run a complete model-A-vs-model-B analysis: bootstrap CIs, permutation test, honest verdict
- Diagnose flawed statistical claims and name the specific error in each
| Spaced-rep warm-up: the full Week 9 due deck | 10 min |
| Guided: closed-book formula sheet drill | 18 min |
| Guided: 20-question cumulative assessment + grading | 37 min |
| Practice: flawed-claims clinic | 15 min |
| Project: model A vs model B mini-lab + verdict | 30 min |
| Quiz + map misses to revisit days | 10 min |
Builds on: Day 57 — Probability & Monte Carlo · Day 60 — Sampling, CIs & bootstrap · Day 61 — Hypothesis tests & power · Day 62 — Entropy & cross-entropy
A mechanic doesn't inspect a toolkit by reading the labels on the drawers — they pull each tool out and use it on something. Today is inspection day for the sharpest week of math in the program. No new tools; instead you close the notes and find out which handles your hand actually reaches for and which drawers are secretly empty.
The science here is boring and bulletproof: retrieving something from memory strengthens it far more than re-reading it, struggling before checking beats checking first, and finding a gap today — while the material is a week old — is cheap, whereas finding it on Day 139 in front of a real eval report is expensive. So the day has three movements: write the formulas cold and diff against your notes; take a 20-question assessment that mixes all six days so your brain has to pick the right tool, not just reuse the one from the current chapter; then do the job this week was training you for all along — look at two models' eval scores and decide, with error bars and a test, whether A actually beats B. That decision is the single most repeated statistical act of an AI engineer's career.
Week 9 is the statistical spine of everything downstream: Day 75's metrics, Day 102's sampling, Day 139's eval statistics, Day 140's capstone eval report. The interleaved assessment matters because real problems never announce their chapter — "is this retrieval change better?" doesn't tell you it wants a paired test with CIs. And the A-vs-B mini-lab is a dress rehearsal for the exact deliverable Day 140 grades: a quality claim with error bars that survives a skeptical reviewer.
Guided practice
The closed-book formula sheet
18 min- Close every note and editor. On paper or a blank file, write from memory: conditional probability, Bayes' rule (with the normalizer spelled out), binomial mean and variance, standard error of a mean, the 95% CI for a pass rate, the permutation-test recipe (as 3 numbered steps), entropy, cross-entropy, KL as a difference, and the random-baseline loss for k classes.
- Now open your notes and diff line by line. Mark each item green (exact), yellow (right shape, wrong detail), or red (blank/wrong).
- For every yellow and red: write a fresh flashcard in your own words, and say OUT LOUD one sentence about where the formula earns money (e.g. "pass-rate CI → every eval report").
- Re-write the reds from memory once more before moving on — the second retrieval is where the repair happens.
The 20-question assessment
37 min- Take the assessment in one sitting, closed-book, ~90 seconds per question. Pull 20 questions from the week's quiz banks and flashcard decks — 3–4 per day from Days 57–62 — or have your AI tutor generate them from the day titles, mixed in random order.
- Coverage checklist (verify before starting): conditional probability & the complement trick (D57), binomial noise & CLT & heavy tails (D58), Bayes & base rates (D59), CIs & bootstrap & coverage (D60), p-values & power & peeking (D61), entropy & cross-entropy & perplexity (D62).
- Grade honestly. 16+/20: proceed. 12–15: fine — this is exactly what today is for. Below 12: schedule tomorrow's first 30 minutes for the two weakest days before starting Week 10.
- For every miss, write the day number to revisit and ONE sentence on why the wrong answer tempted you — the temptation is the misconception, and naming it is the cure.
- Update your error log (the Day 28 artifact): statistical misconceptions belong in it just like algorithm bugs.
On your own
The flawed-claims clinic
15 minFive claims cross your desk. For each, name the specific error and write the one-sentence correction:
- "Our fraud model is 99% accurate, so 99% of its alerts are real fraud." (base rate: 0.2%)
- "Model B scored 35/40 vs A's 33/40 — B is better, shipping it."
- "p = 0.20, so there's an 80% chance our change worked."
- "Mean latency is 80 ms, well under our 100 ms target — we're fine."
- "Loss hit 2.3 on our 10-class classifier — great progress from 4.1!"
Hints: the errors are, in some order — confusing P(E|H) with P(H|E); ignoring binomial noise on a small n; misreading a p-value as a posterior; using a mean on a heavy-tailed quantity; missing the ln(k) baseline. Match them, then check against Days 58–62.
Mini-lab: does model A beat model B?
Run the week's capstone decision. The starter recipe: with rng = np.random.default_rng(seed=63), simulate 60 shared eval questions with per-question difficulty d = rng.uniform(0.05, 0.6, size=60), then case-level results a = rng.random(60) > d and b = rng.random(60) > d * 0.82 (B is genuinely a bit better). Produce ab_verdict.md: pass rates with 95% bootstrap CIs (import your Day 60 bootstrap.py), a permutation test on the gap (your Day 61 ab_test.py), a paired disagreement table (A-only-right vs B-only-right counts), and a three-sentence stakeholder verdict naming effect size, uncertainty, and what evidence would settle the question. Commit it — Day 140's eval report is this document at production scale.
Common mistakes & misconceptions
- Re-reading notes instead of retrieving. Recognition feels like knowledge; only closed-book recall reveals (and repairs) the gaps.
- Taking the assessment open-book "just to check". You are measuring retrieval strength; an open book measures your ability to read.
- Skipping the miss-to-revisit-day mapping. An ungraded mistake is a mistake scheduled to repeat on Day 139.
- Running the A/B mini-lab unseeded, getting different numbers each run, and shrugging. Reproducibility is part of the deliverable.
- Comparing A and B only as independent samples when they answered the SAME questions — the paired view uses the shared difficulty and is much sharper.
- Writing a verdict with certainty the data cannot support. "B is better, p = 0.04, effect +7 points (CI 1–13)" is honest; "B wins" is not.
Q1. A screening test is 95% sensitive with a 2% false-positive rate; the condition affects 1 in 500 people. A person tests positive. Roughly how likely do they have the condition?
Q2. Which change narrows a confidence interval on an eval pass rate?
Q3. Your language model's cross-entropy loss is 10.82 nats on a 50,000-token vocabulary. What has it learned so far?
Go deeper — curated resources
- McNemar's test — the formal version of the paired disagreement table — The disagreement-count comparison you did has a name and an exact test. Day 139 uses this family for paired eval comparisons; reading it now makes that day trivial.
- Formula sheet written from memory; reds re-drilled and turned into flashcards
- Assessment scored ≥ 16/20, or a targeted revisit plan written for the weak days
- All five flawed claims corrected with the specific error named
- ab_verdict.md committed with CIs, permutation test, paired table, and an honest verdict
← Back: Every tool today came from this week: Monte Carlo (Day 57), binomial noise (Day 58), Bayes (Day 59), bootstrap CIs (Day 60), permutation tests (Day 61), and the ln(k) baseline (Day 62). The mini-lab imports your own Day 60 and 61 code.
Forward →: Week 10 starts tomorrow with NumPy in anger — the same arrays, pointed at data instead of dice. The A-vs-B verdict you wrote today reappears at production scale on Day 139 (eval statistics) and Day 140 (the capstone eval report with CIs).