Statistics I — Sampling & Confidence
- Distinguish a population parameter from a sample estimate and name common sources of sampling bias
- State the law of large numbers and compute a standard error as sd/√n
- Construct a 95% confidence interval and interpret it correctly (coverage, not certainty)
- Build a bootstrap confidence interval for any statistic with NumPy resampling
- Attach honest error bars to an eval pass rate and say whether two scores are distinguishable
| Spaced-rep warm-up: Days 57–59 cards (conditional prob, CLT, Bayes) | 10 min |
| ELI5 + tech read, walk the sampling-ci visualizer | 20 min |
| Guided: coverage experiment + bootstrap from scratch | 40 min |
| Practice: error bars on the 34/40 vs 31/40 eval delta | 20 min |
| Project: bootstrap.py error-bar machine | 20 min |
| Quiz + flashcards | 10 min |
Builds on: Day 57 — Monte Carlo & the 1/√n law · Day 58 — Distributions, CLT & standard error
A chef never drinks the whole pot to find out if the soup needs salt. One spoonful — IF the pot is well stirred — tells the story. Statistics is the discipline of spoonfuls: the pot is the population (every request your API will ever serve, every question users will ever ask your model), the spoonful is your sample, and the entire game is knowing how far a spoonful can mislead you.
Two things can go wrong. The pot might not be stirred — you tasted only the top, where the fat floats. That is sampling bias, and no amount of math repairs it. Or the pot is stirred but the spoon is small — a tiny taste is noisy, and salt levels wobble from spoonful to spoonful. That wobble is quantifiable: it shrinks like 1/√n, and a confidence interval is just an honest label on the spoon that says "the pot is probably within this range of what I tasted." The bootstrap, today's power move, is delightfully weird: you re-taste your OWN spoonful thousands of times (resampling it with replacement) to measure how wobbly spoonfuls of that size are — no formula required.
Every number you will report as an AI engineer is a spoonful: an eval pass rate on 40 cases, a latency p95 from an hour of traffic, a thumbs-up rate from 200 users. Report "87% vs 84%, we improved!" without error bars and you will ship regressions with confidence — on a 40-case set, that gap is well inside the noise. Day 139 builds the eval-statistics discipline on today's foundations, and Day 140's capstone eval report requires CIs. Customers and interviewers both ask the same question: "how sure are you?" — today you learn to answer with a number.
Tasting the soup — how confidence intervals narrow as n grows
step 1 / 5Somewhere out there is a TRUE value — the whole pot of soup. Say the true mean is 50. We can never taste the whole pot; we sample spoonfuls.
Guided practice
What "95%" actually means — a coverage experiment
20 min- Paste the starter. It creates a population of 1,000,000 latencies with a KNOWN true mean — a luxury you never have in real life, which is exactly why simulation is the right teacher.
- Draw 1,000 independent samples of n = 50. For each, compute mean, se = sd/√n, and the interval mean ± 1.96·se.
- Count what fraction of the 1,000 intervals contain the true mean. It should land near 0.95 — that fraction IS the meaning of "95% confidence".
- Rerun with n = 200. Coverage stays ~95% but intervals get ~2× narrower (√4). Coverage is the promise; width is what you buy with sample size.
- Now sabotage it: sample only from the slowest 20% of the population (biased sampling) and watch coverage collapse to ~0. Write the lesson: the CI machinery assumes a fair spoonful — bias breaks it silently.
The bootstrap from scratch
20 min- The starter gives you ONE sample of 200 latencies — your real-world situation: no population, no formula for the p95's standard error.
- Read the bootstrap loop:
rng.integers(0, n, size=(10_000, n))builds 10,000 resample index sets at once; fancy indexing turns them into resamples; one statistic per row. - Compute bootstrap 95% CIs for the mean, the median, and the p95. Note how much wider the p95's interval is — tail statistics are hungry for data.
- Cheat and check: the code holds a hidden population. Compare each CI against the true values. The mean and median CIs should cover; the p95 usually covers but is wide.
- Shrink your sample to n = 25 and rebuild the p95 CI. Watch it become absurd — a p95 from 25 points is barely more than the maximum of 25 points. Write the rule: the bootstrap quantifies the noise in YOUR sample; it cannot substitute for data you never collected.
On your own
Error bars or it didn't happen
20 minYour model passed 34/40 golden-set cases yesterday and 31/40 today after a prompt tweak. A teammate wants to revert the tweak.
Your goals: (1) compute the 95% CI for each day's pass rate using se = √(p̂(1−p̂)/n); (2) bootstrap the same CIs from the case-level boolean arrays (simulate them: 40 Bernoulli draws each at the observed rates, or just build arrays with 34 and 31 Trues) and confirm the formula and the bootstrap roughly agree; (3) state whether the two days are distinguishable and what you would tell the teammate; (4) compute how many cases n you would need for the CI to be ±3 points at p ≈ 0.85 — solve 1.96·√(0.85·0.15/n) = 0.03.
Hints: the two intervals overlap heavily — the honest answer is "we cannot tell from 40 cases". The n you compute in (4) is why Day 140's golden set has a minimum size and why Day 139 uses paired comparisons to squeeze more signal from the same cases.
bootstrap.py — your error-bar machine
Promote the guided code into bootstrap.py in your practice repo: a reusable bootstrap_ci(data, stat_fn, n_boot=10_000, alpha=0.05, seed=0) that works for any vectorizable statistic, plus a pass_rate_ci(k, n) helper implementing the formula CI for eval scores. Include a demo section that (a) reproduces the 34/40 vs 31/40 analysis with a printed verdict, and (b) bootstraps a CI for the p95 of a seeded log-normal latency sample. Docstrings state the assumptions (representative sample, independence, n not tiny). Commit it — Day 63's checkpoint imports this file, and Day 139 rebuilds it for real eval runs.
Common mistakes & misconceptions
- Reading "95% CI" as "95% probability the truth is inside this interval". The truth is fixed; the interval is the random thing. 95% is the long-run capture rate of the recipe.
- Quoting an eval score without n. "85%" from 40 cases is 74–96%; from 4,000 cases it is 84–86%. The number alone is meaningless.
- Using standard deviation where standard error belongs. sd describes the data's spread; se = sd/√n describes the estimate's wobble.
- Bootstrapping a biased sample and trusting the result. The bootstrap replays YOUR sample's composition — bias in, bias out, now with respectable-looking error bars.
- Cranking n_boot to narrow an interval. More resamples only smooth the CI estimate; interval width comes from n, the real sample size.
- Bootstrapping tail statistics (p95, max) from tiny samples. A p95 needs enough points in the tail to say anything; check n before trusting the interval.
Q1. A model passes 34/40 eval cases (85%). Roughly what is the 95% confidence interval for its true pass rate?
Q2. What does "95% confidence" technically promise?
Q3. You raise n_boot from 1,000 to 1,000,000 resamples. What happens to your bootstrap CI?
Go deeper — curated resources
- Wilson score interval — better CIs for proportions near 0 or 1 — The normal-approximation CI misbehaves when p̂ is extreme or n is small (it can exceed 100%). The Wilson interval fixes this and is what serious eval harnesses use — worth knowing before Day 139.
- Coverage experiment shows ~95% at two sample sizes and collapse under biased sampling
- Bootstrap CIs computed for mean, median, and p95 with the vectorized index trick
- Eval-delta practice answered: CIs, verdict, and required n for ±3 points
- bootstrap.py committed; quiz ≥ 2/3
← Back: The 1/√n wobble you eyeballed on Day 57 is now the standard error, and Day 58's CLT is why "estimate ± 1.96·se" is allowed to use the normal's 95% number.
Forward →: Day 61 asks the follow-up question — is this difference real? — with hypothesis tests. Day 63's checkpoint runs bootstrap CIs on a model-A-vs-model-B decision, and Day 139 turns today's toolkit loose on real eval runs (with pairing tricks that beat naive CIs).
Unlocks: D61 Statistics II — Hypothesis Tests & A/B · D63 Week 9 Checkpoint: Math Assessment · D74 Ensembles — Forests & Boosting · D76 Validation, Bias/Variance & Regularization