Observability Complete
- Recall the Week-21 observability stack from memory: gates, traces, flywheel, red-team, drift, dashboard
- Verify the capstone observability gate: every request traced, feedback live, gate in CI, dashboard populated
- Close or ticket every open red-team finding and promote flywheel candidates into the golden set
- Re-quiz the week's weak spots and refresh the spaced-repetition deck
| Spaced-rep warm-up: full Week-21 due deck | 10 min |
| Guided: blank-page recall of the observability loop | 15 min |
| Guided: operate every instrument once from memory | 25 min |
| Practice: cumulative re-quiz + deck refresh | 15 min |
| Project: pass the observability gate + write the report | 40 min |
| Quiz + finalize the punch-list for Phase 8 | 15 min |
Builds on: Day 141 — Regression gates & CI for AI · Day 142 — Tracing LLM applications · Day 144 — Red-teaming lab · Day 146 — Quality & cost dashboards
A pilot does not take off because the plane "feels ready." They run a checklist, and every instrument must read green: fuel, hydraulics, controls, radios. Green is not a mood; it is each gauge confirming the thing it watches is actually working. Only when the whole panel is green does the plane roll.
This is a review day, and its analogy is the pre-flight scan: your docs-QA service now has a full instrument panel, and today you confirm every light is green before the deployment phase begins next week. The confirmation is deliberately hands-on — you don't re-read the lessons, you *operate* the instruments from memory. Trigger a trace and read it back. Post a thumbs-down and watch it land as a joinable log. Sabotage a prompt and watch the CI gate block the merge. Fire the red-team suite and check the findings are closed or ticketed. Open the dashboard and read this week's quality, latency, and cost. Spaced repetition works here because you're not memorizing facts — you're rehearsing a checklist until running it is automatic, which is exactly what pre-flight is: the same scan, every single flight, until it's muscle.
Week 21 is the hinge of the whole capstone: Phase 8 (deployment) is only safe on top of observability. Shipping a docs-QA service you cannot trace, gate, or measure is how teams get the 2am page with no data to debug it. The review exists because a stack assembled piece by piece across six days has seams — the gate that passes locally but not in CI, the trace that drops spans under async load, the dashboard wired to stale logs. An FDE presenting to a customer's engineering leadership will be asked, live, to show each instrument working; rehearsing that demo today, from memory, is the difference between a confident cutover and a scramble.
Guided practice
Blank-page recall: draw the observability loop
15 min- Close all your notes and the editor. On a blank page, draw the Week-21 loop from memory: request → trace → logs+feedback → continuous eval/drift → dashboard → (on failure) flywheel → eval case → gate; with the red-team suite feeding the same gate.
- On each arrow, write the one key decision it carries (e.g. trace → "tail-bias sampling, redact PII"; gate → "block on floor, confirmed across 3 runs").
- Now open your capstone and diff your drawing against what actually exists. Every arrow you couldn't draw, or that isn't wired, is today's punch-list item.
- Write the gaps into
observability_punchlist.md.
Operate every instrument once, from memory
25 minRun the pre-flight scan hands-on. For each instrument, perform the action and confirm the reading:
- Trace: fire one query, grab the returned trace ID, and read the full span tree back with your Day-142 viewer. Green = correctly parented spans with cost/tokens on the LLM span.
- Feedback: post a thumbs-down via curl and confirm it lands in the logs joined to that request ID.
- Gate: sabotage the system prompt on a branch, open a PR, and confirm the CI eval gate blocks it; restore and confirm it passes. terminal:
git checkout -b test-gate && ... - Red-team: run the attack suite and confirm ASR is reported and every non-zero finding is closed or ticketed.
- Drift + dashboard: run the aggregator and open
dashboard.html; read this week's faithfulness, p95 latency, and cost aloud as if to a stakeholder.
Record each instrument's status (green/amber/red) in the punch-list.
On your own
Cumulative re-quiz + deck refresh
15 minTake the cumulative Week-21 quiz below (it spans all six days). For every question you miss, open the day named in its revisit pointer, re-read only the relevant section, and rewrite that day's weakest flashcard in your own words. Then run your due spaced-repetition cards for the week and mark honestly — a card you hesitated on is a card that isn't learned. The goal is not a score; it's a ranked list of what to shore up before Phase 8.
Hints: if you miss the drift question, you likely conflated online eval with the offline golden set — re-draw the distinction. If you miss the gate question, revisit the block-vs-warn rule and the version triple.
Week-21 checkpoint: pass the observability gate
Bring the capstone to "instruments all green." Produce observability_report.md that demonstrates, with real artifacts, each gate criterion: (1) a sample trace ID resolved to its span tree; (2) a feedback record joined to its request; (3) a screenshot or log of the CI eval gate blocking a sabotaged prompt and passing when restored; (4) the dashboard populated from this week's real telemetry, with the weekly-review note; (5) the red-team ASR report with every finding closed or ticketed (owner + due date). End with a one-paragraph readiness statement: is the service observable enough to deploy, and what one gap (if any) you're carrying into Phase 8 with a mitigation. This report is the artifact you'd show a customer's engineering lead.
Common mistakes & misconceptions
- Re-reading the week instead of operating it. Recognition is not recall; the checkpoint tests whether you can run each instrument, so drill by doing, not reading.
- Declaring green without CI proof. A gate that passes locally but was never run on a real PR is not wired — prove it blocks on the platform, not your laptop.
- Leaving red-team findings "known but open" with no ticket. An untracked finding is a forgotten finding; every non-zero ASR result gets closed or ticketed with an owner.
- A dashboard wired to last week's logs. Confirm the aggregator reads current telemetry, or the cockpit shows a green that no longer exists.
- Skipping the blank-page recall because the code exists. The interview and the customer demo ask you to reconstruct the design from memory — find the gaps now.
- Treating the carried gap as a failure to hide. Naming one honest gap with a mitigation is what a mature readiness statement does; hiding it is how it bites in production.
Q1. A prompt PR passes the eval gate locally but you never opened a pull request. Is the observability gate satisfied?
Q2. For a wrong-answer bug report, which Week-21 instrument tells you first whether retrieval or generation failed?
Q3. Quality slips over two weeks with no deploy in the git log. Which instrument is designed to catch this, and how?
Go deeper — curated resources
- Rehearse the customer observability demo — Script a 5-minute walkthrough of your instrument panel for a non-technical stakeholder, then a 5-minute version for an engineering lead — Day 167 grades exactly this demo craft.
- Observability loop drawn from memory and diffed against the capstone
- All five instruments operated hands-on with status recorded
- observability_report.md demonstrates every gate criterion with real artifacts
- Red-team findings closed or ticketed; flywheel candidates reviewed
- Cumulative quiz ≥ 2/3 and due deck cleared
← Back: This checkpoint consolidates Days 141–146 and closes the Phase-7 arc that began with the Day-134 eval mindset and the Day-140 harness; it also revisits Day 136's retrieval-vs-generation split as the first trace question.
Forward →: Phase 8 opens on Day 148 by containerizing this observable service; Day 152 folds the CI gate into a full deploy pipeline, and Day 157 graduates the dashboard into formal SLOs — none of which is safe without the green panel you confirm today.