Week 18 Checkpoint: Support-Triage Agent
- Reconstruct the agent loop, planning harness, memory system, patterns menu, and guard structure from memory
- Assemble the Week 18 modules into a support-triage agent: classify → retrieve → draft → escalate-with-approval
- Measure the agent on a 20-ticket labeled set and report classification accuracy, grounding, and escalation precision/recall
- Score ≥ 2/3 on the cumulative Week 18 quiz and requeue weak spots
| Spaced-rep: full due deck for Week 18 (SM-2 drill) | 15 min |
| Guided 1: blank-page recall + diff + new cards | 20 min |
| Guided 2: assemble and run the triage shift | 40 min |
| Practice: worst-ticket error analysis | 15 min |
| Project wrap: README metrics + tag week18 | 15 min |
| Cumulative quiz + requeue missed topics | 15 min |
Builds on: Day 120 — Agents I — the loop · Day 122 — Agents III — memory & context · Day 123 — Workflow patterns — routing · Day 125 — Agent reliability
The intern trained all week: Monday the work loop, Tuesday planning, Wednesday the notebook system, Thursday how the firm organizes teams, Friday the safety rules. Today is her first real shift — a morning inbox of twenty support tickets — and, just as important, her certification exam. First the exam, because a skill you can't recall without notes isn't yours yet: you'll rebuild the week's machinery from a blank page, and every stumble becomes a flashcard. The stumbles are the point — memory science says the recalls that almost fail are the ones that stick hardest.
Then the shift. For each ticket she: reads it and decides what KIND of problem it is (the front desk from Thursday), looks up the relevant policy in the company library (last week's open-book system, now one of her tools), drafts a reply that cites its sources, and — the judgment call — decides whether this one is above her pay grade. Refund over the limit? Security scare? Furious customer? Those go to a human, with her notes attached. And because it's her first shift, you don't grade on vibes: twenty tickets with known right answers, counted. How many classified correctly, how many replies actually grounded in the docs, how many escalations correct — and which mistake is cheapest.
This checkpoint is the program's thesis in miniature: RAG (Week 17) and agents (Week 18) composing into one measured system — precisely the architecture behind real support-automation products, and the single most common "design an AI system" interview brief. The evaluation half is the deeper lesson: "measured on a 20-ticket set" is the difference between a demo and a claim you can defend, and it previews Phase 7 (Days 134–140), where evals become your daily discipline. The artifact also feeds forward directly: Day 137 evals this exact agent's trajectories, and Day 160's system-design round asks you to whiteboard what you will have actually built today.
An intern with a to-do list — the agent loop, step by step
step 1 / 5Task: "Find why order #4412 failed and draft a reply." The agent holds a goal, a scratchpad, and tools.
Guided practice
Blank-page recall: rebuild Week 18 from memory
20 minClose every editor tab and note first.
- Write the Day 120 agent loop skeleton from memory — decide/act/observe, errors-as-observations, the four stop conditions. Then diff against
agent/core.pyline by line. - List the five workflow patterns with a one-phrase mechanism each, and state the workflow-vs-agent distinction (D123).
- Draw the memory architecture: three memory types, what compaction keeps verbatim, and the O(N²) cost argument (D122).
- Write the guard taxonomy table — four failure classes, detector for each (D125) — and the three MCP primitives with who-decides (D124).
- Reproduce the plan-harness contract: what a done_when is, the failure ladder, who owns state (D121).
- Diff everything; every gap becomes a flashcard. Re-answer this week's two hardest quiz questions from the dashboard's missed list.
The shift: assemble and run the triage agent
40 min- In the practice repo, create
triage/. Writetickets.json: 20 tickets you author — 6 billing (2 with refund amounts over the 100 USD limit), 5 bug reports (1 mentioning "security breach"), 6 how-to/policy questions answerable from your Nimbus capstone corpus, 3 account requests (1 from a twice-angry customer thread). Label each: category, expected_docs (section id or null), escalate (true/false). Authoring the labels IS designing the eval — you did this for the corpus on Day 119. - Write
triage/agent.pywiring the week's modules per the architecture you committed to inagent_patterns.md(D123): router → (RAG tool for docs tickets) → draft → evaluator gate → escalation check → approval queue. Run everything insideGuardedtoolsets with the audit trail on. The scripted classifier/drafter stand-ins keep it runnable in-browser; the README maps each to its real LLM call (you have made this mapping four times this week — it should feel mechanical). - Write
triage/score.py: read the audit trail + outputs, compute classification accuracy, grounding rate, escalation precision/recall, and print the 4×4 confusion table. - Run the shift. Twenty tickets, one command, three numbers and a table out.
- The security-breach ticket and both over-limit refunds MUST be in the approval queue. If any slipped through, that is a failed guard — find it in the audit trail before touching any code.
On your own
Error analysis: the worst ticket
15 minTake your worst failure (a misclassification, an ungrounded reply, or a missed escalation — if you scored perfectly, harden a ticket until you don't: add an ambiguous billing-bug hybrid, a policy question using vocabulary far from the corpus, or a polite message hiding a refund demand). Diagnose it with the right week's tool: routing errors → D123 (router confidence, category definitions); grounding failures → walk the Day 118 RAG flowchart; escalation misses → D125 (rule coverage vs classifier confidence). Fix at the guilty layer, rerun the full 20, and append a 3-line postmortem to the README: symptom → layer → fix → metric before/after. One number must improve and none may regress — if a fix trades accuracy for recall, say so explicitly.
The measured triage agent
Deliver triage/ complete in the practice repo: 20 labeled tickets, the assembled agent (router + RAG tool + evaluator-gated drafting + escalation approval queue + memory + guards), the scorer, and a README reporting the three headline metrics with the confusion table, the postmortem, and the real-LLM mapping for every scripted stand-in. Targets: classification ≥ 16/20, grounding ≥ 80%, escalation recall = 100% on the three must-escalate tickets (precision may pay for it — say what it cost). Tag it week18. This is the artifact Day 137 evals, and the system you will diagram from memory in the Day 160 design round.
Common mistakes & misconceptions
- Grading the agent by reading a few transcripts. Twenty labeled tickets with code-computed metrics is the minimum honest claim; vibes-grading is how demos ship and products fail.
- Tuning escalation for precision. The false-negative (missed security incident) costs incomparably more than a human reviewing an unnecessary escalation — tune for recall and report the price.
- Rebuilding instead of composing. Today's agent is five imports and wiring; if you are writing new loop logic, stop and use the week's modules — composition is the checkpoint skill.
- Letting the drafter answer doc questions from its own head. The RAG tool exists so replies cite chunks; an eloquent uncited reply about the refund policy is the capstone's named nightmare in agent clothing.
- Reviewing by rereading lesson notes. Blank-page reconstruction first — the diff IS the review; recognition without recall decays in days.
- Skipping the confusion table. Aggregate accuracy hides that all four errors are billing-vs-account swaps — the table tells you which prompt or rule to fix; the average tells you nothing.
Q1. The triage agent's escalation gate should be tuned toward recall because…
Q2. A reply about the refund policy is fluent, plausible, and cites nothing. Under this week's architecture, which component failed?
Q3. Ticket 14 was processed but you cannot tell whether the RAG tool was called before drafting. What is missing?
Go deeper — curated resources
- Trajectory vs outcome scoring — Today you scored outcomes. Day 137 also grades the PATH — did it retrieve before drafting, escalate before replying? Your audit trail already contains everything that eval needs; skim it with that question in mind.
- Blank-page recall diffed; gaps converted to flashcards
- Twenty tickets processed in one command inside guards with a complete audit trail
- Three headline metrics + confusion table computed by code and reported
- All must-escalate tickets in the approval queue; postmortem written
- Cumulative quiz ≥ 2/3 (retake after targeted review if lower)
← Back: Composition day: D120's loop, D121's verification, D122's memory and budget, D123's routing and evaluator gate, D124's tool interface, D125's guards — plus Week 17's retriever as the tool that grounds it all.
Forward →: Day 127 pivots to adaptation (fine-tuning); Day 132 attacks this agent with injection; Day 137 evals its trajectories from today's audit trails; Day 160 asks you to whiteboard this architecture under interview pressure.
Unlocks: D137 Agent & Task Evals