Bayes' Rule
- Write Bayes' rule from memory and name prior, likelihood, evidence, and posterior
- Solve the medical-test problem with natural frequencies and explain why P(disease|+) ≪ test accuracy
- Identify base-rate neglect in real product and debugging claims
- Build a tiny naive Bayes spam classifier from word counts and explain each factor
- Apply Bayesian updating to a debugging/incident-triage scenario
| Spaced-rep warm-up: Day 57–58 cards (conditional probability, binomial) | 10 min |
| ELI5 + tech read, walk the bayes-grid visualizer | 20 min |
| Guided: medical test three ways + naive Bayes filter | 42 min |
| Practice: Bayesian incident triage | 18 min |
| Project: base-rate briefing memo | 22 min |
| Quiz + flashcards | 10 min |
Builds on: Day 57 — Conditional probability & Monte Carlo · Day 58 — Distributions & expectation
You hear a fire alarm in your building. Do you believe there is a fire? Probably not — you have heard a hundred alarms and seen zero fires. Your belief starts from experience (fires are rare), gets nudged by evidence (the alarm), and lands somewhere in between: "almost certainly burnt toast, but let's glance at the hallway." Now add smoke under the door and your belief flips. You just ran Bayes' rule twice, natively, in your head.
Bayes' rule is the arithmetic of changing your mind. You start with a prior — how plausible the idea was before the new clue. You ask how loudly the clue shouts for the idea — the likelihood: alarms happen often without fires, so an alarm shouts quietly; smoke rarely appears without fire, so smoke shouts loudly. Multiplying "how plausible before" by "how loud the clue" and rescaling gives the posterior — your new belief. The classic human bug is ignoring the prior: a 99%-accurate test for a one-in-a-thousand disease still yields mostly false alarms, because the disease was so rare to begin with. Rare things stay fairly rare even after impressive-sounding evidence.
Bayes shows up on both sides of your future job. Inside the models: spam filters, classifier probabilities (Day 72's logistic regression outputs are posteriors), and the whole "P(next token | context)" framing of language models. Inside your head: incident triage is Bayesian — "deploys break things far more often than cosmic rays" is a prior that routes 80% of debugging correctly (Day 172 makes this a formal method). And when a customer says "your model flagged 200 frauds, how many are real?", the answer is a base-rate calculation — get it wrong and you torch trust in the system you just sold.
Updating your beliefs — 1,000 people take a 90%-accurate test
step 1 / 6| test + | test − | total | |
|---|---|---|---|
| sick | ? | ? | 10 |
| healthy | ? | ? | 990 |
The PRIOR: a disease affects 1% of people. In a town of 1,000, that's 10 sick, 990 healthy — before anyone is tested.
Guided practice
The medical test three ways: counts, formula, simulation
20 min- Paste the starter. Part A computes P(disease|+) via natural frequencies (a 100,000-person table). Predict the answer before running — most people guess near 99%.
- Part B computes the same thing with the Bayes formula. Confirm the two agree (~9%).
- Part C is the Monte Carlo check: simulate a million patients, filter to positives with a boolean mask, take the mean of disease among them — Day 57's conditional-probability move.
- Now experiment: raise prevalence to 10% (screening a high-risk group) and rerun. Watch the posterior jump to ~92%. Same test, different prior, wildly different meaning — write down why doctors order tests only when symptoms raise the prior.
- Lower the false-positive rate to 0.1% at 0.1% prevalence. The posterior hits ~50%. Note which knob mattered more here — the false-positive rate fights the prior directly.
A naive Bayes spam filter in 40 lines
22 min- The starter ships a tiny labeled corpus (10 spam, 10 ham messages) inline — no downloads.
- Read the
trainfunction: it counts word frequencies per class with Laplace add-one smoothing. Say out loud what each count estimates (it is P(word|class)). - Read
classify: it SUMS LOG probabilities instead of multiplying. Comment out the log version and multiply raw probabilities for a 60-word message to see why (tiny numbers → underflow toward 0.0). - Run the classifier on the three test messages and inspect the per-class log scores.
- Break it on purpose: classify a message full of words the training set has never seen. Confirm smoothing keeps it from crashing to a hard zero, and that the PRIOR (spam vs ham counts in training) decides the tie.
- Add two spam and two ham messages of your own invention to the corpus and re-run. Watch a word's spam-score shift as its counts change — that is "updating on evidence" made mechanical.
On your own
Bayesian incident triage
18 minYour API's error rate just spiked. From six months of postmortems you know the base rates of root causes: bad deploy 60%, provider outage 25%, traffic spike 10%, database issue 5%. You observe one clue: the provider status page is green (assume: P(green | provider outage) = 0.2 — status pages lag; P(green | any other cause) = 0.95).
Your goals: (1) compute the posterior over all four causes given the green status page (normalize across all four hypotheses — this is Bayes with more than two hypotheses); (2) state which cause to investigate first and second; (3) a second clue arrives — no deploy happened in the last 24h (P(no-deploy | bad deploy) = 0.05, P(no-deploy | others) = 0.7). Update AGAIN, using your first posterior as the new prior. (4) Write the two-line lesson about chaining evidence.
Hints: build a dict of priors, multiply each by its likelihood, divide by the sum. Sequential updating = yesterday's posterior is today's prior.
The base-rate briefing memo
Write bayes_memo.md (plus a small bayes_calc.py helper): a one-page memo to a fictional customer whose fraud model "is 99% accurate" on transactions with a 0.2% fraud base rate. Compute the honest precision of an alert (use natural frequencies and show the 100,000-transaction table), propose the two levers that would actually improve alert quality (raise the prior by pre-filtering; lower the false-positive rate), and include a reusable posterior(prior, sensitivity, false_pos) function with three worked calls. Written for a smart non-statistician — the Day 105 and Day 171 stakeholder-communication muscle starts here.
Common mistakes & misconceptions
- Confusing P(E|H) with P(H|E). "99% of sick people test positive" does not mean "99% of positives are sick" — the base rate decides the gap.
- Ignoring the prior entirely (base-rate neglect). Impressive evidence about a rare hypothesis usually still leaves it unlikely.
- Forgetting the normalizer when comparing multiple hypotheses — posteriors must sum to 1 across ALL candidate causes, not just the two you like.
- Multiplying many raw probabilities in code. They underflow to zero; always sum log-probabilities.
- Letting one unseen word zero out a naive Bayes score. Laplace (add-one) smoothing exists precisely to prevent hard zeros from sparse counts.
- Updating on the same evidence twice — re-reading the same status page is not new information. Each update must use genuinely new evidence.
Q1. Disease prevalence 0.1%, sensitivity 99%, false-positive rate 1%. Roughly what is P(disease | positive test)?
Q2. In naive Bayes, why sum log-probabilities instead of multiplying probabilities?
Q3. During triage you compute a posterior over causes, then a second independent clue arrives. What do you do?
Go deeper — curated resources
- The prosecutor's fallacy — P(evidence|innocent) vs P(innocent|evidence) — Courtroom cases (Sally Clark) where this exact confusion convicted innocent people. The strongest motivation you will find for keeping the direction of conditioning straight.
- Medical-test posterior computed three ways, all agreeing (~9%)
- Spam filter runs; underflow experiment and unseen-word experiment done
- Triage exercise: two-step sequential update computed correctly
- Briefing memo committed; quiz ≥ 2/3
← Back: Bayes is Day 57's conditional probability read in reverse — the simulation in Part C is literally a boolean-mask filter. The log-sum trick leans on logarithm intuition from Day 53.
Forward →: Day 62's cross-entropy is built from the same log-probabilities. Day 72's logistic regression outputs posteriors; Day 75's precision/recall is the base-rate problem industrialized; Day 172 turns today's triage exercise into a formal debugging method.
Unlocks: D62 Entropy, Cross-Entropy & KL · D75 Evaluation Metrics · D135 Graders — Code, Rubric & LLM-as-Judge