Statistics for Evals
- Explain why a single eval run is a noisy sample, not a fixed truth
- Compute a confidence interval on a pass rate with the bootstrap, in runnable Python
- Use paired comparisons to detect small but real differences between two systems
- Estimate the minimum sample size to distinguish a target effect
- Set regression thresholds that separate real drops from run-to-run noise
| Spaced-rep: due cards (human eval, judges) + recall Day 60 bootstrap | 10 min |
| ELI5 + tech read; why one eval run is a noisy sample | 20 min |
| Guided: bootstrap CI + paired comparison | 45 min |
| Practice: adjudicate a real eval delta | 20 min |
| Project: statistical rigor for the eval report | 15 min |
| Quiz + flashcards | 10 min |
Builds on: Day 60 — Sampling, standard error & bootstrap CIs · Day 61 — Hypothesis tests & A/B pitfalls · Day 134 — Golden sets & pass rates
You score two prompts on your eval: A gets 82%, B gets 85%. Ship B, right? Not yet. Those percentages are measured on a sample of, say, 40 questions — and if you'd picked 40 slightly different questions, or the model had sampled tokens a little differently, the numbers would wobble. The 85% is not a fact; it is a dart that landed at 85, thrown by a slightly shaky hand. The real question is whether the hand is shaky enough that 82 and 85 are the same throw.
Error bars answer that. Instead of "85%", you report "85%, give or take 6" — and once you see that A's range (76–88) and B's range (79–91) overlap heavily, "B is better" evaporates. Maybe it is; you just can't tell from 40 questions. This is the discipline that separates people who move numbers from people who move NOISE and congratulate themselves. Every eval result is a measurement with uncertainty, and a result reported without its uncertainty is half a result — often the wrong half.
This is where most eval work quietly falls apart: teams celebrate a prompt change that moved the score from 82% to 85% on 40 cases, ship it, and the "improvement" was noise — next week it's 81%. An engineer who reports eval deltas with confidence intervals, uses paired tests to find real effects, and knows their 40-case set can't distinguish a 3-point difference is dramatically more credible than one waving a bare percentage. For the capstone's regression gate (Day 141), you MUST separate a real quality drop from run-to-run jitter, or the gate either blocks good changes or waves through bad ones. "Error bars or it didn't happen" is the whole professional posture.
Tasting the soup — how confidence intervals narrow as n grows
step 1 / 5Somewhere out there is a TRUE value — the whole pot of soup. Say the true mean is 50. We can never taste the whole pot; we sample spoonfuls.
Guided practice
Bootstrap a confidence interval on your pass rate
20 min- Create
eval_stats/bootstrap.pyfrom the starter. Take a real vector of per-case pass/fail (1/0) from your Day 134 eval run (or the provided example). - Implement the bootstrap: resample the results with replacement B=10,000 times, recompute the mean each time, and take the 2.5/97.5 percentiles for a 95% CI.
- Print "pass rate X% (95% CI: L–U)". Compare the CI width at n=20, 40, 100, 400 by truncating/replicating your data — watch the interval shrink like 1/√n.
- Also compute the normal-approximation interval and compare; note where they diverge (small n, p near 0/1).
- Write one sentence: given your actual set size, what is the smallest quality difference you could even claim to detect?
Paired comparison of two systems
25 min- Create
eval_stats/paired.py. You have per-case pass/fail for system A and system B on the SAME golden cases (starter provides vectors; or use your real before/after run from Day 134). - Compute each system's marginal pass rate and its bootstrap CI. Notice the CIs overlap — the naive read says "no difference."
- Now go paired: bootstrap the per-case DIFFERENCE (B−A), and report the delta's mean and 95% CI. Because shared case-difficulty cancels, this CI can exclude 0 even when the marginal CIs overlap.
- Add McNemar's exact-ish check: count cases where A passed but B failed (b) vs B passed but A failed (c); the effect lives entirely in the discordant pairs. Print b, c and the implied direction.
- Write the honest verdict: is B's improvement real at 95%, or is the delta CI straddling 0 (inconclusive — you need more cases, per the sample-size note)?
On your own
Adjudicate a real eval delta honestly
20 minTake a real before/after from your capstone (a prompt or chunking change from Day 134/136) and write eval_stats/verdict.md. Report: each version's pass rate WITH a bootstrap CI, the paired delta with its CI, the discordant-pair counts, and a one-line ship/don't-ship/inconclusive verdict that follows from the statistics — not from the point estimate.
Then answer the uncomfortable question: given your current set size and its CI half-width, what is the minimum improvement you could actually detect, and how many more cases would you need to detect a 3-point gain? (Rough it: to halve the CI, quadruple n.) If the answer is "I can't conclude anything from 40 cases," write that — it is the correct, senior answer.
Hints: an improvement whose paired-delta CI includes 0 is not an improvement you can claim. "Inconclusive, need ~4× the cases" beats a confident lie.
Statistical rigor for the capstone eval report
Upgrade your capstone's evals/RESULTS.md (and add evals/eval_stats/) so every reported number carries uncertainty. Requirements: (1) every pass rate / mean-faithfulness / win-rate reported with a bootstrap 95% CI; (2) any A-vs-B claim backed by a paired analysis with the delta's CI and discordant counts, not two overlapping marginal CIs; (3) a stated minimum detectable effect for your current golden-set size, and the n you'd need for a 3-point resolution; (4) a proposed regression-gate threshold (for Day 141) chosen to sit OUTSIDE the noise band, with the reasoning. If your system is stochastic, run each case ≥ 3 times and report across-run variance. This turns your eval from a number into a defensible measurement.
Common mistakes & misconceptions
- Reporting a bare pass rate. A number without a CI hides its sampling error; readers over-trust a point estimate that could swing 12 points on a re-sample.
- Declaring B > A from overlapping marginal CIs — or from a bare 82% vs 85%. Use a PAIRED comparison; shared case difficulty cancels and reveals (or refutes) the real effect.
- Over-reading a small set. 40 cases can only distinguish large effects; claiming a 3-point win from 40 cases is noise worship. State the minimum detectable effect.
- Regression gates tighter than the noise band. If the CI half-width is 10 points, failing on a 3-point drop cries wolf constantly and trains everyone to ignore the gate.
- Ignoring run-to-run non-determinism. Temperature/sampling make repeats differ; for stochastic systems, average multiple runs and report across-run variance too.
- Confusing statistical significance with importance. A real but tiny gain may not be worth a cost/latency regression — significance is necessary, not sufficient (Day 61).
Q1. Prompt A scores 82% and prompt B 85% on the same 40-case eval. Before shipping B you should…
Q2. Why does a PAIRED comparison detect smaller true differences than comparing each system's independent confidence interval?
Q3. Your eval's 95% CI half-width is about ±10 points. A sensible regression gate for CI (Day 141) fails the build when the score drops by…
Go deeper — curated resources
- McNemar's test and paired proportions — For paired pass/fail, the effect lives entirely in the discordant pairs (A-pass/B-fail vs B-pass/A-fail). McNemar's test formalizes when their imbalance is significant — the classic tool for before/after on the same eval set.
- Bootstrap CI computed on a real pass rate; CI shrinkage with n observed
- Paired delta CI + discordant counts computed for a real A/B change
- Verdict written that follows the statistics (including "inconclusive" when true)
- Eval report upgraded so every metric carries a CI; regression threshold proposed outside the noise band
- Quiz ≥ 2/3
← Back: This is Day 60's bootstrap and standard error and Day 61's A/B discipline applied to the pass rates from Days 134–137. The base-rate humility echoes Day 59.
Forward →: Tomorrow (Day 140) every capstone eval number ships with a CI. Day 141's regression gate uses the noise-band threshold you set here, and Day 145's drift monitoring watches these intervals move over time in production.
Unlocks: D140 Week 20 Checkpoint: Capstone Eval Harness · D141 Regression Gates & CI for AI · D145 Drift & Continuous Eval in Prod