Logging, Feedback & the Data Flywheel
- Design structured request logs that make every production interaction replayable
- Build a feedback capture path (explicit + implicit) joined to requests by ID
- Mine logs and feedback for new eval cases with a prioritized sampling strategy
- Run the flywheel loop: log → cluster failures → new case → fix → gate
| Spaced-rep warm-up: gate + tracing cards | 10 min |
| ELI5 + tech read: logs, feedback, mining, the loop | 20 min |
| Guided: feedback endpoint + log miner | 45 min |
| Practice: implicit-signal design + retry detector | 15 min |
| Project: close one full flywheel loop | 20 min |
| Quiz + flashcards | 10 min |
Builds on: Day 138 — Human evaluation & feedback UX · Day 141 — Regression gates · Day 142 — Tracing LLM applications
When one aircraft hits unusual turbulence, the report doesn't stay with that pilot. It flows into a shared system, gets clustered with similar reports, and next month every pilot in the fleet gets a briefing and a revised checklist. One plane's bad afternoon becomes the whole fleet's improvement. Airlines got safe not by hiring perfect pilots but by refusing to waste a single incident.
Your docs-QA service can work the same way. Every question a user asks is a flight; every thumbs-down, retry, or abandoned session is a turbulence report. Left alone, those reports evaporate. Captured and joined to the flight recording (yesterday's traces), they become the most valuable dataset you own: real users, real phrasing, real failures your golden set never imagined. The flywheel is the loop that turns them into permanent improvement: a bad answer becomes an eval case, the eval case fails, you fix the cause, and the Day-141 gate ensures that particular failure can never quietly return. Each turn of the wheel makes the next turn easier — that's why it's a flywheel and not a hamster wheel.
Golden sets age. The questions you invented on Day 140 are not the questions users actually ask, and the gap widens every week. Teams with a working flywheel improve weekly on real traffic; teams without one keep polishing performance on an imaginary distribution. This is also the honest answer to the interview question "how do you improve an LLM product after launch?" — not "better prompts," but "a pipeline from production failures to regression-gated eval cases." For an FDE, showing a customer their own users' failure clusters, with fixes gated in CI, is what turns a pilot into a renewal.
Every flight teaches the fleet — the data flywheel, one revolution
step 1 / 5Production serves thousands of requests. Each leaves a structured log: prompt, retrieved context, answer, latency — and, crucially, user feedback. The raw ore of improvement.
Guided practice
Request logs + a feedback endpoint on the capstone
25 min- Add the logging middleware and
/feedbackendpoint from the starter code to your capstone FastAPI app (adapt names to your project layout). Reuse the request ID you already return asX-Trace-ID. - Apply the Day-142
redact()function to the query before logging. - Start the API and fire 5 varied questions at
/ask. terminal:curl -s localhost:8000/ask -X POST -H 'content-type: application/json' -d '{"question":"what is the pto policy"}'— note the request ID in the response headers. - Post feedback for two of them, one negative with a category: terminal:
curl -s localhost:8000/feedback -X POST -H 'content-type: application/json' -d '{"request_id":"PASTE_ID","rating":"down","category":"missing_source"}'. - Check
requests.jsonl: confirm request records and feedback records share IDs and that queries are redacted.
Mine the logs for eval candidates
20 min- Generate traffic: run 20 questions against the capstone (mix easy, hard, ambiguous, and 3 deliberately out-of-scope), and post feedback on at least 8, several negative.
- Write
mine_cases.pyfrom the starter: join feedback to requests, apply the priority sampler, and group negatives by category and shared keywords. - Run it and inspect the candidate file: each candidate must carry the real (redacted) query, the bad answer, the doc IDs retrieved, and a blank
expected:field for you to fill. - Fill in
expectedfor 3 candidates and promote them intoevals/candidates.jsonl— Day 147 reviews them into the golden set.
On your own
Design the implicit-signal catcher
15 minExplicit feedback rates are typically under 5%, so design the silent majority's signals. In writing (no code yet): define detection rules for (1) a retry — same user asks a near-duplicate question within 2 minutes; (2) an abandonment — client disconnects mid-stream; (3) a success proxy of your choice. For each: how you detect it from existing logs, its false-positive story (when the signal lies), and the weight you would give it vs an explicit thumbs-down when prioritizing mining. Then implement just the retry detector against your requests.jsonl.
Hints: near-duplicate = high cosine similarity of query embeddings or normalized string overlap; a follow-up question is NOT a retry — that is the false-positive story.
Close one full loop, for real
Turn one production failure into a permanent regression guard, end to end, and document each station. Pick the worst genuine failure from today's mined negatives (or manufacture a realistic one by asking something your corpus covers badly). (1) Capture: show the log record and trace. (2) Case: write it as a proper eval case with expected behavior. (3) Reproduce: show the harness failing on it. (4) Fix: adjust prompt, retrieval, or chunking — smallest change that works. (5) Gate: show the case passing and committed into the canary or full suite, with the Day-141 gate green. Write flywheel.md in the capstone repo narrating the five stations with the actual artifacts (log line, case JSON, before/after harness output), plus a standing weekly triage checklist.
Common mistakes & misconceptions
- Logging without stable IDs. Feedback that cannot be joined to its request, trace, and prompt version is a sentiment survey, not data.
- Promoting production failures straight into the golden set without review. Users downvote correct answers too; unreviewed cases poison the exam (Day 134).
- Building a five-field feedback form. Response rate collapses; two clicks maximum, categories optional, freeform last.
- Treating each thumbs-down as its own bug. Cluster first — five complaints about the same missing document are one retrieval fix.
- Logging raw queries and comments verbatim. Feedback text is user data; redact before write, same as traces.
- Running the flywheel once and declaring victory. It is a ritual, not a feature — without the scheduled weekly triage the wheel stops within a month.
Q1. Why must the request ID travel to the client and come back with feedback?
Q2. A user asks nearly the same question again 40 seconds after getting an answer. Best interpretation?
Q3. What is the correct path from a production failure to "this can never regress again"?
Go deeper — curated resources
- Failure clustering with embeddings — Embed 100+ logged queries (Day 92 skills), k-means them (Day 78), and name the clusters. Compare against your keyword grouping — which surfaces a pattern you missed?
- Feedback posted via curl appears in logs joined to its request
- Miner run produces reviewed candidates with provenance
- One failure carried through all five flywheel stations, gate green
- flywheel.md committed with real artifacts and the weekly triage checklist
- Quiz ≥ 2/3
← Back: Day 138 designed the human side of feedback; Day 142's traces and IDs make it joinable; Day 141's gate is the station that makes each fix permanent.
Forward →: Day 145 samples these same logs for continuous online evals, Day 146 charts feedback rate and pass trends on the dashboard, and Day 147 reviews your candidate cases into the golden set.
Unlocks: D145 Drift & Continuous Eval in Prod · D146 Quality & Cost Dashboards