Production Debugging with Customers
- Run the triage framework — impact → scope → recent changes — before touching any code
- Perform log and trace forensics on an LLM application to isolate the failing stage
- Choose between hotfix and root-cause fix deliberately, and schedule the one you skipped
- Communicate during an incident in a way that lowers the customer's blood pressure
- Convert a closed incident into regression tests, eval cases, and runbook entries
| Spaced-rep warm-up: due cards from Days 169–171 | 10 min |
| ELI5 + tech read: triage and forensics | 20 min |
| Guided: three scenarios + log forensics lab | 40 min |
| Practice: incident comms | 18 min |
| Project: runbook v2 with five playbooks | 22 min |
| Quiz + flashcards | 10 min |
Builds on: Day 142 — Tracing LLM applications · Day 158 — Reliability & incident response · Day 170 — Customer environments
A field mechanic gets called to a farm where the harvester died mid-season. The farmer is losing money by the hour and hovering at the mechanic's elbow. A rookie starts unbolting panels immediately — looks busy, learns little. The veteran asks three questions first: What exactly stopped working — engine, cutter, or conveyor? (impact). Does the other harvester do it too? (scope). What changed since it last worked — new fuel, new operator, yesterday's storm? (recent changes). Three questions in, the search space has collapsed from "the whole machine" to "the fuel line," and the farmer has calmed down, because someone with a method is on the case.
Debugging production AI with a customer watching is the same craft under the same pressure. The three questions come before any code. The engine's flight recorder — your traces from Day 142 — replaces guesswork with a timeline. And the talking IS part of the fix: a calm "here's what we know, here's what we're checking, next update at 3" is worth as much to the customer as the patch itself.
"The AI is giving wrong answers" is the vaguest incident report in software, and you will receive it constantly — often about an environment you cannot touch (Day 170). LLM apps fail in stages (ingest, retrieval, rerank, prompt, model, parsing), non-deterministically, and sometimes because a PROVIDER changed something under you. FDEs live at this exact intersection: the customer sees you debug, so your method — or your flailing — becomes their opinion of your entire company. Interviewers probe it directly: "walk me through a production incident you handled."
Guided practice
Three scenarios, one framework
20 minFor each scenario below, write a "first 15 minutes" plan BEFORE reading on: your impact/scope/recent-changes questions, the first data you'd pull (dashboard panel? traces? diagnostic bundle from Day 170?), and your top two hypotheses ranked.
Scenario A — "Answers got worse yesterday." Meridian's champion emails: users say the assistant "started making things up" since Wednesday. Your deploy log shows nothing shipped. Think: what ELSE changes without you shipping? (Two candidates from Day 145 — provider model update, corpus ingest run — and Wednesday's ingest log shows 40 new documents.)
Scenario B — "It's unusable at 9 a.m." The VPC deployment (their kitchen, your app) times out every weekday 9:00–9:30, fine otherwise. You cannot SSH in. Think: what is special about 9 a.m.? (Everyone logs on; token-refresh stampede; shared gateway rate limits; a scheduled ingest job colliding with peak traffic.)
Scenario C — "It showed me another department's document." An HR user saw a snippet from a Legal-restricted file. Think: is this an ACL-sync gap (Day 169), a metadata-filter bug, or a cache serving another user's answer (Day 156)? What makes this one different from A and B? (It is a security incident: different comms, different urgency, preserve evidence.)
Then compare against the reference plans at the bottom of the starter file for the lab below — score yourself: did you quantify impact first? Did "recent changes" include non-deploy changes?
Log forensics lab
20 minScenario A made concrete. The starter code contains 12 synthetic trace summaries (JSON lines) from before and after Wednesday's ingest — the shape your Day 142 instrumentation actually emits.
- Create
forensics_lab.pywith the starter and run it — it just parses and pretty-prints per-stage data. - Extend it to answer, with numbers: (a) mean
groundednessbefore vs after Wednesday; (b) whether retrieval latency or the retrieved-chunk sources changed; (c) which SOURCE the bad answers' chunks come from. - Diagnosis: the post-Wednesday failures all cite chunks from
hr_policy_2026_DRAFT.docx— a draft uploaded into the watched folder, contradicting the final policy. Retrieval works perfectly; the CORPUS is poisoned. Write the one-paragraph diagnosis naming the failing stage (ingest governance, not the model). - Write the two-line hotfix (exclude the draft, re-index) and the root-cause list (ingest filters for draft/temp files, a source-quality check in the eval set, a runbook entry).
On your own
Incident comms under pressure
18 minWrite the three customer-facing messages for Scenario A (the poisoned draft document), addressed to Meridian's champion with the CISO cc'd:
- Acknowledgement — send within 15 minutes of the report: what you know (facts only, no cause speculation), what you're doing, when the next update lands. ≤ 90 words.
- Update at T+2 h — cause found (draft doc in the watched folder — phrase it without blaming THEIR admin who uploaded it), hotfix live, what remains. ≤ 120 words.
- Closure at T+2 days — cause, fix, prevention (ingest filter + new eval case + runbook entry), in plain exec language (Day 171). ≤ 150 words.
Constraints: never write "the model hallucinated" (unfalsifiable and wrong here); the closure must make the customer MORE confident than before the incident — name the machinery that now catches this class automatically.
Symptom → playbook runbook v2
Upgrade the capstone runbook (started Day 154, completed Day 161) with a customer-facing debugging section: five symptom-driven playbooks — (1) "answers got worse", (2) "it's slow / timing out", (3) "user saw something they shouldn't", (4) "provider outage / rate limited", (5) "ingest failed or corpus stale". Each playbook: first three triage questions, exact data to pull (dashboard panel name, trace query, or diagnostic-bundle command from Day 170 — real commands, not descriptions), top 3 causes ranked with the discriminating test for each, hotfix lever (flag/rollback/pin), and comms template pointer. Add the Scenario A regression: an eval case asserting no DRAFT-flagged source is ever cited. Commit; Day 177's quality gates check this file exists and is real.
Common mistakes & misconceptions
- Opening the code before quantifying impact. Ten minutes of dashboard and trace reading collapses the search space more than an hour of reading source — and calms the customer, because method is visible.
- Forgetting that "recent changes" includes changes you did not make: provider model updates, customer-side uploads, expiring certificates, a new proxy. Your deploy log being empty rules out almost nothing.
- Trusting one trace of a non-deterministic system. Sample several bad and several good requests before declaring a stage guilty; one weird trace is weather, five is climate.
- Blaming "the model" by default. In RAG apps the model is the LAST suspect: corpus, retrieval, filters, and prompt assembly fail far more often — and the traces can prove it either way.
- Hotfixing and moving on. A hotfix without a scheduled root-cause fix, regression test, and eval case is a time bomb with your name on it; Day 143's flywheel exists to metabolize incidents.
- Going silent while you investigate, or speculating in writing. Silence breeds catastrophic customer theories; written speculation gets quoted back at you. Promise a cadence, keep it, state facts.
Q1. Users report worse answers since Wednesday; you deployed nothing. Per the triage framework, the FIRST question to investigate is:
Q2. Traces show retrieval returning the correct chunks, but answers contradict them. The failing stage is most likely:
Q3. A user saw content from a document their role cannot access. What makes this scenario different from a quality bug?
Go deeper — curated resources
- Blameless postmortem for Scenario A — Write the full postmortem using Day 158's template: timeline, contributing factors (the watched-folder design, not the person who uploaded), action items with owners. Notice how "who uploaded the draft" becomes irrelevant.
- First-15-minutes plans written for all three scenarios before reading references
- Forensics lab pinpoints the draft document with numbers, not vibes
- Three incident messages written within the word limits
- Runbook v2 committed with 5 playbooks + draft-source eval case; quiz ≥ 2/3
← Back: Day 142's traces are the flight recorder you just read; Day 158 taught the internal incident machinery — today added the customer in the room. Day 145 predicted the "nothing deployed, everything changed" class of incident.
Forward →: Day 174's simulation includes a mid-engagement fire drill. Day 177 verifies the runbook and the new eval case as quality gates, and interviewers on Day 179 will ask for exactly the incident story you practiced today.