Day 158 · When the kitchen catches fire

Reliability & Incident Response

You will be able to

Enumerate the failure modes specific to LLM apps: provider outages, rate limits, bad deploys, quality regressions
Implement timeouts, bounded retries with jitter, and a circuit breaker around model calls
Design a fallback chain — secondary model, cached answer, honest degradation — and defend its order
Run an incident with roles, a timeline, and calm comms, then write a blameless postmortem
Rehearse a provider-down scenario against your capstone before it happens for real

Today's ~120 minutes

Spaced-rep warm-up: due cards incl. Day 46 idempotency & Day 157 SLOs	10 min
ELI5 + tech read; sketch the fallback chain from memory	15 min
Guided: resilient wrapper + tabletop exercise	45 min
Practice: blameless postmortem	20 min
Project: incident runbook	20 min
Quiz + flashcards	10 min

Builds on: Day 46 — Distributed systems — retries & idempotency · Day 125 — Agent reliability — graceful degradation · Day 157 — Monitoring & SLOs — the alarms that start incidents

The analogy

🧸 ELI5⚙ Tech

Professional kitchens do not prevent all fires; they rehearse for them. Every cook knows the drill: who grabs the extinguisher, who moves the guests, who calls it in, and — crucially — the restaurant does not stop serving. The menu shrinks to what the working stations can produce. Cold dishes go out. Nobody stands in the dining room shouting "EVERYTHING IS BROKEN"; someone calmly tells guests there will be a short delay on hot mains.

Production AI systems catch fire on a schedule you don't control: your model provider has an outage, you get rate-limited during your own launch spike, a deploy ships a subtle regression. Reliability engineering is the rehearsed drill: timeouts so one stuck call can't hold a table forever, retries that give up gracefully instead of hammering a downed provider, a circuit breaker that stops sending cooks into a burning station, and a fallback menu — a backup model, a cached answer, or an honest "I can't answer right now, here's a link to the docs." The postmortem afterward asks "what let the fire spread?" never "which cook do we blame?" — because a kitchen that hides mistakes burns down twice.

Why this matters on the job

Your LLM app has a hard dependency on someone else's API, and every major provider has had multi-hour incidents — the question is when, not if. Customers judge you by minute twenty of an outage: did the app degrade honestly or return raw stack traces? Did someone communicate? FDEs are on the sharpest end — Day 172 has you debugging DURING a customer incident, and the difference between "we exercised this fallback last month" and improvising live is the difference between renewal and churn. Interviews probe this too: "what happens when the model API goes down?" is a standard Day 160 follow-up.

Guided practice

guided 1

Build the resilient call wrapper

25 min

Create resilient.py from the starter: timeout, bounded retries with jittered backoff, circuit breaker, and a fallback chain, wrapped around two simulated flaky providers.
Run the demo loop: provider A fails 100% for a stretch. Watch the log: retries → circuit opens → calls fail fast to provider B → half-open probe → recovery.
Tune: set retries to 5 with no backoff and re-run. Count how many useless calls hammer the dead provider before the circuit opens — this is the retry-storm anti-pattern made visible.
Now integrate the pattern into your capstone's model-call path with a real chain: primary model → cheaper backup model → retrieval-only degraded answer. Ensure each degraded response sets a degraded: true field and increments the Day 157 fallback counter.

🐍 python — editable, runs in your browser

import random, time

class CircuitBreaker:
    def __init__(self, fail_threshold=3, cooldown_s=5.0):
        self.fails, self.threshold = 0, fail_threshold
        self.open_until, self.cooldown = 0.0, cooldown_s

def allow(self):
        return time.monotonic() >= self.open_until

def record(self, ok: bool):
        if ok:
            self.fails = 0
        else:
            self.fails += 1
            if self.fails >= self.threshold:
                self.open_until = time.monotonic() + self.cooldown
                self.fails = 0
                print("  [breaker] OPEN — failing fast for cooldown")

def call_with_retries(fn, breaker, attempts=2, base=0.2):
    if not breaker.allow():
        raise RuntimeError("circuit open")
    for i in range(attempts):
        try:
            out = fn()
            breaker.record(True)
            return out
        except Exception:
            breaker.record(False)
            if i + 1 < attempts and breaker.allow():
                time.sleep(base * (2 ** i) * (0.5 + random.random()))
    raise RuntimeError("exhausted retries")

def answer(question, providers, breakers):
    for name, fn in providers:
        try:
            return call_with_retries(fn, breakers[name]), name
        except RuntimeError:
            continue
    return "Sorry — I can't generate an answer right now. "            "Top matching docs: [handbook §4, policy §2]", "degraded"

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Tabletop exercise: your provider is down

20 min

Run this solo tabletop against your capstone, writing the incident log in real time (timestamps included). Scenario script — reveal one step at a time:

T+0 — Day 157's canary fails twice; fallback-rate alert pages you. Primary provider returns 529s. Write your first three actions in order.
T+5m — Provider status page confirms a major incident, no ETA. Decide: wait, or flip traffic to the backup chain? Write the decision AND the one-line stakeholder update you post (audience: your fictional users' Slack channel).
T+20m — Backup model is up but 30% slower and slightly weaker; fallback rate is 100%, latency SLI is breaching. Does the latency SLO page you into a second incident? What do you tell users now?
T+90m — Provider recovers. Define your recovery checklist: how do you confirm health before flipping back (hint: half-open probe, canary green, error rate), and what do you monitor for the next hour?
Compute the total error-budget burn from the incident using Day 157's numbers, and note whether the budget survived.

On your own

Write the blameless postmortem

20 min

Turn the tabletop into a one-page postmortem in docs/postmortems/2026-08-provider-outage.md with sections: Summary, Impact (requests affected, budget burned), Timeline (from your log), Contributing causes (find at least three — the outage itself, plus what made it worse on YOUR side), What went well, Action items (owner + date each).

Constraints: no names blamed (it's you anyway — blame the system that let a single provider be a single point of failure); every action item must be verifiable ("add second provider to chain" not "be more careful").

Hints: good candidate causes — no pre-warmed backup, canary interval too long, no status page for users.

Ship before you stop

The capstone incident runbook

Write docs/runbook.md — the document future-you opens at 3am: (1) service overview with the fallback chain diagrammed in text; (2) "alarm → first moves" table for your top five alerts (canary red, fallback spike, latency breach, error spike, cost spike), each with diagnosis commands and mitigation; (3) rollback procedure (Day 154, verified); (4) provider-outage play from today's tabletop; (5) comms templates — the three-sentence status update and the all-clear; (6) postmortem template. Keep every play executable by a stressed person: numbered commands, no prose walls. This completes the runbook Day 154 started and is required for Day 161's gate.

Rubric — check what you completed (0/5)

Fallback chain implemented in the capstone with degraded:true labeling and metricsRunbook covers all five alarms with copy-pasteable diagnosis and mitigation stepsRollback procedure tested this week and marked with the test dateTabletop incident log and completed postmortem committedComms templates present and under four sentences each

Common mistakes & misconceptions

Retrying everything. 4xx errors (bad request, auth) will fail identically forever; retry only 429/5xx/timeouts, with backoff AND jitter, or you synchronize a retry storm.
No circuit breaker, so every user request eats the full retry-timeout ladder against a dead provider. Fail fast once failure is established; probe for recovery automatically.
A fallback chain that silently degrades. Unlabeled cached or weaker-model answers destroy trust when discovered; set degraded flags, show users honest banners, count it in metrics.
Fallback to a same-vendor model tier and calling it redundancy. Shared infrastructure fails together; true redundancy crosses providers (and ideally regions).
Diagnosing before mitigating. Users are down while you read logs; rollback or flip to fallback first, investigate on a healthy service.
Postmortems that end with "root cause: human error, action: be careful." Find the systemic causes and write verifiable action items with owners, or the incident repeats.

Knowledge check

Q1. Your provider starts returning 529 (overloaded). Naive clients everywhere retry immediately in a tight loop. What pattern-pair prevents your service from making the outage worse?

Q2. During degraded mode, why is "here are the top matching documents" often the right fallback for a RAG app?

Q3. A postmortem concludes: "Cause: the engineer typo'd the config. Action: engineers will double-check configs." What is wrong?

Go deeper — curated resources

bookGoogle SRE Book — Ch. 15: Postmortem Culture ↗25 min bookGoogle SRE Book — Ch. 4: SLOs (error-budget policy) ↗15 min repoSystem Design Primer — availability patterns & circuit breakers ↗20 min

If you have a third hour

Error-budget policy — The pre-agreed rule for what happens when the budget empties (feature freeze, reliability sprint). Agreeing on it BEFORE the incident is what makes Day 157's budgets enforceable instead of decorative.

Done means

Resilient wrapper integrated: timeouts, jittered retries, breaker, labeled fallbacks
Tabletop completed with a timestamped incident log
Postmortem and runbook committed; rollback re-verified
Quiz ≥ 2/3

How this connects

← Back: Day 46 gave you retries and idempotency; Day 125 taught graceful degradation for agents; Day 154 rehearsed the rollback you reached for; Day 157's alarms are what page you into today's process.

Forward →: Day 161's cutover gate requires the runbook and a tested fallback. Day 172 replays this discipline inside a customer's environment, where calm comms are half the job — and the Day 160 interviewer will ask "what if the provider goes down?" expecting today's answer.

Unlocks: D160 AI System Design · D161 Week 23 Checkpoint: Production Cutover · D172 Production Debugging with Customers