Monitoring & SLOs
- Distinguish metrics, logs, and traces and pick the right one for a given question
- Define SLIs and SLOs for an LLM service and compute the implied error budget
- Design alerts that page on user-visible symptoms, not on every internal blip
- Instrument a FastAPI service with request-rate, error, and latency-histogram metrics
- Add LLM-specific monitors: quality score, refusal rate, token spend, and drift hooks
| Spaced-rep warm-up: due cards incl. Day 142 tracing & Day 146 dashboards | 10 min |
| ELI5 + tech read; recite SLI → SLO → error budget from memory | 15 min |
| Guided: RED instrumentation, SLO math, canary | 50 min |
| Practice: the alert audit | 15 min |
| Project: monitoring pack assembly & drill | 20 min |
| Quiz + flashcards | 10 min |
Builds on: Day 58 — Distributions — heavy-tailed latencies · Day 142 — Tracing LLM applications · Day 146 — Quality & cost dashboards · Day 155 — Latency percentiles for the capstone
A house needs smoke alarms, not a fire marshal standing in every room. A good smoke alarm has three properties: it triggers on real danger (smoke, not toast — mostly), it triggers early enough to act, and it is quiet the rest of the time. An alarm that shrieks daily about toast gets its battery removed, and then the real fire finds a silent house.
Monitoring production software is the same craft. You cannot watch everything, so you choose a handful of signals that stand for user experience — "are requests succeeding? fast enough? at what cost?" — and set thresholds that mean act now. The formal version has three layers: an SLI is the measurement itself (the smoke density sensor), an SLO is the promise you set against it ("99.5% of requests succeed within 3 seconds"), and the error budget is the promise's built-in allowance for failure — the 0.5% you are permitted to burn. When the budget is burning fast, you stop shipping features and fix reliability. When it is barely touched, you are allowed to move fast. The budget turns "is it reliable enough?" from an argument into arithmetic.
"Do you have SLOs?" is a question customers' platform teams will ask an FDE in the security-and-operations review before any enterprise deal closes — right next to the Day 159 checklist. Internally, alert design is what determines whether your on-call weeks are calm or ruinous: teams that page on symptoms sleep; teams that page on causes drown in noise and then miss the real outage. And LLM services add monitors traditional apps never needed — a service can be 100% up, fast, and cheap while quietly answering wrongly. Day 161's cutover gate requires today's alarms wired and tested.
Guided practice
Instrument the capstone with RED metrics
25 min- Add
prometheus-clientto the capstone and mount the starter middleware. Hit a few endpoints, then open/metricsand find your counters and histogram buckets. - Add two LLM-specific metrics: a counter for fallback/refusal responses and a histogram for token spend per request (use the Day 156 ledger values).
- Run Prometheus locally via Docker (or just curl /metrics repeatedly) and confirm numbers move under a small load loop (20 requests).
- Write the PromQL-style expression for: p95 latency over 5 minutes, error ratio over 5 minutes, and fallback rate over 1 hour. Keep them in
docs/slo.md— they become your alert rules.
Write the SLOs and do the error-budget math
15 min- In
docs/slo.md, define three SLIs for the capstone: availability ("2xx or honest-fallback within 8s"), latency ("TTFT under 1.5s"), and quality ("sampled continuous-eval pass"). - Set an SLO for each over 30 days. Be realistic: you are one person on a hobby budget — 99.5% availability is defensible; 99.99% is fiction (write down what 99.99% would allow: ~4 minutes of downtime a MONTH).
- Compute each error budget in absolute terms at 10k requests/month: how many failed requests, how many slow ones, how many bad answers.
- For the availability SLO, compute two burn-rate alert thresholds: a page (budget gone in ~6 hours at current rate) and a ticket (gone in ~3 days). Note which channel each goes to.
Build the synthetic canary
10 min- Write
canary.pyfrom the pattern below: every run, ask production one known golden question, validate the answer contains the expected grounded fact and a citation, and record success/latency (push to a metrics file or a pushgateway). - Schedule it (cron or GitHub Actions schedule) every 5–15 minutes against your staging/production URL.
- Break it on purpose: point the app at an empty index or unset the API key, and confirm the canary fails BEFORE you would have noticed by hand. That confirmation — the alarm fires in a drill — is the whole point.
On your own
The alert audit
15 minYou inherit this alert list for an LLM service: (a) CPU > 80% for 1 min → page; (b) any single 5xx → page; (c) p95 latency > 3s for 10 min → page; (d) daily token spend > budget → page at midnight; (e) fallback rate > 10% for 15 min → page; (f) disk 70% full → page.
For each: keep as page, demote to ticket/dashboard, or rewrite — with one line of justification grounded in "does a human need to act NOW, and is this a symptom users feel?" Produce the corrected list.
Hints: two of these are causes, not symptoms; one is noise by construction; one has the wrong urgency channel; two are close to correct.
Capstone monitoring pack
Ship monitoring to the capstone: (1) the RED + LLM metrics endpoint live in staging; (2) docs/slo.md with three SLIs/SLOs, absolute error budgets, and two burn-rate alert rules each with its channel; (3) the canary scheduled and proven to fail in a drill (screenshot or log of the drill in the doc); (4) the Day 146 dashboard updated with panels for fallback rate, token spend, and canary status. This is half of Day 161's cutover gate — the alarms must exist AND have fired once on purpose.
Common mistakes & misconceptions
- Averaging percentiles. The mean of per-minute p95s is not the p95; record histograms and derive percentiles from buckets at query time.
- Alerting on causes (CPU, memory, disk) instead of symptoms (error rate, latency, fallback rate). Causes belong on dashboards; symptoms decide pages.
- Setting SLOs at 100% — or at whatever the service currently does. A 100% SLO means any single failure is a crisis and no error budget exists; an SLO copied from current behavior is a description, not a promise.
- Treating "the service is up" as "the service is correct." An LLM app needs quality SLIs — sampled evals, groundedness checks, refusal rate — or it can fail for weeks while every infra graph stays green.
- Paging on single events ("any 5xx"). One blip pages a human at 3am for nothing; burn-rate windows page only when the budget is genuinely being consumed.
- Never testing the alarm. An alert that has not fired in a drill is a hypothesis, not a smoke alarm. Break production-adjacent things on purpose, on a schedule.
Q1. Your availability SLO is 99.5% over 30 days at ~100k requests/month. Roughly how many failed requests does the error budget allow?
Q2. Which alert is best designed, per the symptom-vs-cause rule?
Q3. Every infra metric is green, yet users report wrong answers for a week. Which monitor was missing?
Go deeper — curated resources
- Google SRE Book — Ch. 6: Monitoring Distributed Systems ↗ — The "four golden signals" chapter pairs with RED; read it before designing dashboards for anything with queues.
- /metrics live in staging with LLM-specific metrics
- slo.md committed with budgets in absolute numbers and burn-rate rules
- Canary scheduled and its failure drill documented
- Alert audit completed with justifications; quiz ≥ 2/3
← Back: Day 58 explained why latency percentiles, not means; Day 142's traces are what you open when today's alerts fire; Day 145's continuous evals become the quality SLI; Day 146's dashboard gains its alerting layer.
Forward →: Tomorrow (Day 158) is what happens when the pager goes off — incident response and fallbacks. Day 161's cutover requires today's alarms tested; Day 172 uses these signals to debug in customer environments.
Unlocks: D158 Reliability & Incident Response · D161 Week 23 Checkpoint: Production Cutover