Agent & Task Evals
- Distinguish outcome scoring from trajectory scoring and say when each matters
- Evaluate tool-call correctness (right tool, right arguments, right order)
- Build a task-eval harness with a mocked environment for deterministic replay
- Treat cost and latency as first-class eval metrics, not afterthoughts
- Score a multi-step agent run and localize where a failed task went wrong
| Spaced-rep: due cards (RAG eval, judges) | 10 min |
| ELI5 + tech read; trajectory vs outcome framing | 20 min |
| Guided: replayable harness + tool-call/budget checks | 45 min |
| Practice: eval your Day 126 agent | 20 min |
| Project: agent eval harness for the capstone | 15 min |
| Quiz + flashcards | 10 min |
Builds on: Day 126 β Support-triage agent to evaluate Β· Day 134 β Golden sets & the eval loop Β· Day 136 β Decomposing quality into measurable parts
Grading an intern on a multi-step errand is harder than grading a one-line answer. Sometimes you only care about the outcome: did the report end up on the right desk? But often the outcome hides the story. An intern who delivered the right report by luck β after calling the wrong department, ignoring the instructions, and taking four hours β is not someone you trust with the next errand. And an intern who did everything right but hit a locked door at the last step deserves a different conversation than one who wandered off.
So you grade two things. The outcome: was the final result correct? And the trajectory: were the steps sensible β did they pick the right tools, in a reasonable order, with the right inputs, without burning the whole afternoon? A good agent eval watches the whole errand, not just the doorstep. And because errands touch the real world (sending email, hitting APIs), you rehearse them in a fake building β a mocked environment β so the same errand runs the same way every time and you can replay it after a fix.
Your Day 126 triage agent and any agent you ship is judged on task completion, and "it worked when I tried it" is worthless because agents are non-deterministic and touch external systems. Interviewers and customers ask "how do you know your agent works?" and the credible answer involves a replayable harness with mocked tools, outcome AND trajectory scoring, and cost/latency budgets β because an agent that completes tasks correctly but makes 40 tool calls and costs a dollar each is not shippable. Agents fail in more ways than chatbots (loops, wrong tool, right tool wrong args, gives up early), so the eval has to localize the failure, not just flag it.
Guided practice
A replayable agent harness with mocked tools
25 min- Create
agent_eval/harness.pyfrom the starter. It defines a tiny support-triage agent interface and MOCK tools:search_docs(query)andescalate(ticket)that return canned results and RECORD every call. - Define 4 task cases: a ticket that should be answered from docs (no escalation), one that should escalate, one where the doc lookup returns nothing (agent should escalate, not invent), and one adversarial ticket (an injected instruction it must ignore β Day 133 payoff).
- Run the agent against the mocks. For each case record: outcome (final decision correct?), trajectory (was search_docs called before drafting? correct tool for the subtask?), tool-call log, step count, and a fake cost.
- Print a per-case scorecard and a summary: outcome pass rate, trajectory pass rate, mean steps, mean cost. Note any case that passed outcome but failed trajectory β the lucky success.
- Because tools are mocked, re-run twice and confirm identical trajectories (modulo model sampling). This determinism is what makes it a regression suite.
Tool-call correctness and budgets
20 min- Extend the harness with
agent_eval/toolcheck.py: given a recorded trajectory, score tool selection (was the chosen tool the expected one for each subtask?), argument correctness (did search_docs get a query derived from the ticket, not an empty/garbage string?), and ordering (dependencies respected). - Add budgets to each case: max_steps and max_cost. A case PASSES only if outcome is correct AND it stayed within budget. Show a case that is correct-but-over-budget flipping the run to FAIL.
- Introduce a deliberately bad agent variant (calls escalate first, then searches) and confirm trajectory + ordering scores catch it even when the outcome happens to be right.
- Produce a final table: outcome / trajectory / tool-args / within-budget per case. Write two sentences: for THIS agent, is the weakness in decisions (outcome) or in process (trajectory/tools)?
On your own
Eval your Day 126 agent for real
20 minPoint the harness at your actual Day 126 support-triage agent (or the closest thing you built). Run the 20-ticket set it was originally measured on, but now score outcome AND trajectory AND cost/latency, with mocked doc-lookup so it is replayable.
Deliver agent_eval/report.md: outcome pass rate, trajectory pass rate, mean/95th-percentile steps and cost, and β the interesting part β the list of cases that pass outcome but fail trajectory (lucky) or pass trajectory but fail outcome (right process, wrong result). For each of those mismatches, one sentence on what it reveals.
Hints: if outcome and trajectory pass rates are far apart, that gap is the story. Lots of lucky successes = a fragile agent that will regress on new tickets; lots of good-process-bad-outcome = a tool or knowledge problem, not a reasoning one.
Agent eval harness for the capstone (or triage agent)
Add evals/agent_eval/ to the capstone repo: a replayable harness with mocked tools that scores your agentic component (the triage agent, or your capstone's retrieval-augmented answer flow treated as a 1β2 step agent) on outcome, trajectory, tool-call correctness, and cost/latency budgets. Ship: task cases with fixtures and success criteria + budgets, a runner that records full trajectories, scorers for each axis, and evals/agent-report.md summarizing pass rates and the outcome-vs-trajectory gap. At least one case must be an injected-instruction ticket the agent must ignore (Day 133 tie-in), scored as a trajectory/safety failure if obeyed. This harness joins the Day 140 capstone eval suite.
Common mistakes & misconceptions
- Scoring only the outcome. A right answer reached by the wrong path is a fragile success that will regress; trajectory scoring catches it.
- Evaluating against live tools. Non-determinism and side effects (real emails!) make runs unrepeatable β mock the environment so tasks replay identically.
- Ignoring cost and latency. An agent that is correct but takes 30 steps and a dollar per task is not shippable; budget them as pass/fail criteria.
- No failure localization. "Task failed" is not actionable; the trajectory tells you WHICH step (tool selection, args, ordering, recovery) broke.
- Forgetting safety cases. An agent eval without an injected-instruction task misses the failure mode most likely to become an incident (Day 132β133).
- Treating one run as the score. Agents are stochastic; a single pass/fail per case is noisy β Day 139 shows you need repeats and error bars.
Q1. An agent returns the correct final answer, but the trajectory shows it escalated to a human before ever searching the docs, then got the answer by luck. How should the eval score it?
Q2. Why mock the tools in an agent eval?
Q3. An agent passes correctness on every task but a refactor doubled its average tool calls from 5 to 11. A well-designed eval shouldβ¦
Go deeper β curated resources
- Ο-bench and agent benchmarks β Benchmarks like Ο-bench score agents on realistic tool-use tasks with user simulation and state checks. Study how they define success states and mock environments β the same design your capstone harness uses at smaller scale.
- Replayable harness runs the agent against mocked tools deterministically
- Outcome and trajectory scored separately; the gap reported
- Tool-call correctness and cost/step budgets enforced; an over-budget case fails
- agent_eval harness + report committed to the capstone
- Quiz β₯ 2/3
β Back: This evaluates the Day 126 triage agent using Day 134's loop and Day 136's decompose-to-diagnose habit; the injected-instruction case is Day 132β133; structured tool-arg checks reuse Day 110.
Forward β: Day 138 brings in humans where automation can't judge; Day 139 adds repeats and error bars to these noisy agent runs; Day 140 folds outcome + trajectory + RAG scoring into the one capstone harness.