Statistics II — Hypothesis Tests & A/B
- State null and alternative hypotheses for a product or model change
- Define a p-value precisely and list two things it is NOT
- Run a permutation test from scratch with NumPy and read its verdict
- Estimate statistical power by simulation and size an experiment before running it
- Explain how multiple comparisons and peeking manufacture false positives
| Spaced-rep warm-up: Day 60 cards (se, CI, bootstrap) | 10 min |
| ELI5 + tech read: the courtroom, p-values, power | 20 min |
| Guided: permutation test + power simulation | 40 min |
| Practice: the 20-dashboard trap | 18 min |
| Project: ab_test.py + readout template | 22 min |
| Quiz + flashcards | 10 min |
Builds on: Day 58 — Binomial noise & the CLT · Day 60 — Standard errors & confidence intervals
A courtroom does not ask "is the defendant guilty?" — it asks "is the evidence too strong to be explained by innocence?" The defendant is presumed innocent (the null hypothesis: nothing changed, the difference is luck). The prosecution presents evidence (your data). The jury asks: IF the defendant were innocent, how surprising would this evidence be? That "how surprising" is the p-value. If innocence would produce evidence this damning only 2% of the time, the jury convicts — rejects the null.
Two subtleties make the analogy load-bearing. First, "not guilty" is not "innocent": failing to reject the null means the evidence was insufficient, not that there is no effect — maybe you just needed a bigger investigation (more samples: that is power). Second, the standard of proof is chosen BEFORE the trial (the significance level, usually 5%), because a jury that keeps deliberating until it finds the verdict it wants — peeking at the data until significance appears — will convict innocent defendants constantly. Statistics is a courtroom where you are prosecutor, jury, AND the person who profits from a guilty verdict. The rules exist to protect you from yourself.
"Prompt B scored 3 points higher, ship it" is the AI-engineering equivalent of convicting on a hunch — Day 58 showed 3 points on 40 cases is routine noise. Every prompt tweak, model swap, and retrieval change you A/B (Day 109 does exactly this with two prompt variants) is a hypothesis test, whether you run it honestly or not. Day 139 builds eval significance testing on today's machinery, and Day 141's regression gates are automated hypothesis tests in CI. Teams without this discipline oscillate: ship noise, revert noise, ship it again.
Guided practice
A permutation test from scratch
22 min- The scenario: prompt A passed 33/50 golden-set cases; prompt B passed 41/50. The starter builds the two boolean arrays and computes the observed gap (0.16).
- Read the shuffle loop: pool all 100 results, permute, split back into two groups of 50, record the gap. 10,000 shuffles build the null distribution.
- Compute the two-sided p-value: the fraction of shuffled gaps whose absolute value ≥ 0.16. You should land near p ≈ 0.09.
- Interpret honestly: at α = 0.05 you cannot reject the null — a 16-point gap on 50 cases is still explainable by luck often enough to worry. Feel how counterintuitive that is; this is why eyeballing eval deltas fails.
- Rerun the whole experiment pretending each prompt was evaluated on 200 cases with the same rates (132/200 vs 164/200). Watch p collapse to near zero. Same effect, more evidence, different verdict.
- Print the 95% CI on the gap (Day 60's formula) next to the p-value and note they tell one consistent story.
Power — size the experiment before you run it
18 min- Question: prompt B is TRULY 5 points better (0.80 vs 0.85). How often would your test actually detect that?
- The starter simulates 1,000 complete experiments at each n: draw A ~ Binomial(n, 0.80) and B ~ Binomial(n, 0.85), run a fast normal-approximation test on each, count the fraction reaching p < 0.05. That fraction is the power.
- Run for n = 50, 200, 800, 3200. Expect roughly 8%, 25%, 65%, 99% — at n = 50 you would miss a REAL 5-point improvement more than 90% of the time.
- Find the smallest n (bisect by hand) where power ≥ 0.80. This number — around 1,200 per arm for a 5-point gap — is why "just eyeball 20 cases" is not a methodology.
- Repeat for a 15-point true gap (0.70 vs 0.85) and see power ≥ 80% arrive by n ≈ 100. Write the law in your notes: detectable effect size and required n trade off quadratically — half the effect needs 4× the cases.
On your own
The 20-dashboard trap
18 minYour team tracks 20 metrics. A release goes out that changes NOTHING (you will simulate this: both arms drawn from identical distributions).
Your goals: (1) simulate 20 independent A/A tests (same true rate 0.8, n = 500 per arm), test each at α = 0.05, and count "significant" results — repeat the whole thing 200 times and report the average number of false alarms and the probability of at least one; (2) now simulate peeking: one A/A test where you check significance after every additional 100 observations up to 2,000, stopping at the first p < 0.05 — across 500 simulated experiments, what fraction ever "finds" significance? (3) write the two-line policy you would give a team: one pre-registered primary metric, fixed n decided by a power calculation, no early stopping without sequential methods.
Hints: expect ~1 false alarm per 20-metric release and a ~64% chance of at least one (1 − 0.95²⁰); peeking typically inflates 5% to 25%+. These two numbers explain most "mystery regressions" in dashboards.
ab_test.py + the one-page readout
Build ab_test.py in your practice repo: a reusable perm_test(a, b, n_perm, seed) (vectorize the shuffle loop if you can: one big permuted matrix) and a required_n(p_base, min_effect, power=0.8) estimated by simulation. Then write ab_readout.md — a template you will reuse on Days 109 and 139: hypothesis, primary metric, minimum effect of interest, n from the power calculation, result (effect size + 95% CI + p), decision. Fill the template in for the 33/50 vs 41/50 prompt experiment, with the honest verdict ("promising, underpowered — extend to n≈X before shipping"). Commit both.
Common mistakes & misconceptions
- Reading p = 0.03 as "97% chance the improvement is real". The p-value conditions on the null being true; it says nothing directly about P(H₀).
- Treating "not significant" as "no difference". With low power, real effects routinely fail to reach significance — report the CI so the reader sees what you could and could not detect.
- Confusing significance with importance. With n = 100,000, a 0.1-point difference is "significant" and still not worth shipping. Effect size decides importance.
- Testing 20 metrics and celebrating the one that lit up. At α = 0.05, one in twenty lights up by design. Pre-register a primary metric or correct the threshold.
- Peeking: checking significance repeatedly and stopping when it appears. This inflates false positives severalfold. Fix n in advance.
- Ignoring that both prompts ran on the SAME cases. Paired analysis (per-case comparison) is more powerful than treating the arms as independent — Day 139 exploits this properly.
Q1. A test on your prompt change returns p = 0.03. What does that number mean?
Q2. You monitor 20 metrics after a no-op release, each tested at α = 0.05. What should you expect?
Q3. Prompt B beat prompt A by 6 points on 30 cases, p = 0.4. The most honest conclusion is…
Go deeper — curated resources
- Sequential testing & always-valid p-values — If peeking is inevitable (it is, dashboards exist), sequential methods (SPRT, confidence sequences) let you monitor continuously without inflating error rates. Worth a skim before you build Day 146's dashboards.
- Permutation test built from scratch; p ≈ 0.09 at n=50 and ~0 at n=200 reproduced
- Power curve computed; required n for a 5-point effect estimated
- A/A false-alarm and peeking simulations run with written policy
- ab_test.py + readout committed; quiz ≥ 2/3
← Back: The null distribution is Day 57's Monte Carlo applied to a shuffled world, and the CI you report next to p is Day 60's. The binomial noise that makes 3-point gaps meaningless is Day 58's sd ≈ 2.3 cases.
Forward →: Day 109 A/Bs two prompt variants on 20 cases — you now know why 20 is only a smoke test. Day 139 adds paired comparisons and run-to-run variance for evals, and Day 141 turns significance thresholds into CI regression gates.
Unlocks: D63 Week 9 Checkpoint: Math Assessment · D109 Prompt Engineering II — Memos That Survive Contact · D139 Statistics for Evals