Day 125 · Trust, but verify

Agent Reliability

You will be able to

Name the major agent failure classes (loops, drift, tool misuse, runaway cost) and a detector for each
Implement loop detection, cost guards, and an approval gate as tool-layer wrappers, not prompt requests
Design least-privilege toolsets and idempotent actions for a given agent task
Produce an audit trail (JSONL) sufficient to reconstruct any run after the fact

Today's ~120 minutes

Spaced-rep: due cards + the three MCP primitives from memory (D124)	10 min
ELI5 + tech read: taxonomy and control structure	20 min
Guided: guards firing + least-privilege toolsets	42 min
Practice: red-team and patch	20 min
Project: agent/guards.py + RELIABILITY.md	20 min
Quiz + flashcards	8 min

Builds on: Day 120 — Agents I — the loop · Day 122 — Agents III — memory & context · Day 44 — Security, authN & authZ

The analogy

🧸 ELI5⚙ Tech

The intern did great work all week, so on Friday you're tempted to hand over your corporate card, the master keys, and say "handle everything." No manager on earth does that — not because the intern is bad, but because trust is built with *structure*: give her keys only to the rooms today's job needs (least privilege), require a sign-off before anything irreversible — sending the email, deleting the folder (approvals), have her log every action in the site diary as she goes (audit trail), and agree on a spending cap and a check-in time so a stuck task can't quietly burn the whole day (budgets and loop detection).

Notice what these have in common: none of them are advice to the intern. They are properties of the BUILDING — the doors, the forms, the logbook. You don't ask the intern to please not enter the server room; the badge doesn't open it. Agent reliability is exactly this move: stop trying to prompt your way to safety ("be careful!") and start building the guardrails into the tool layer, where the model physically cannot bypass them. The agent stays creative inside a room you made safe to be creative in.

Why this matters on the job

Agents fail in ways demos never show: retrying the same broken call forty times at 3am, "fixing" a config by deleting it, drifting from summarize-the-tickets to answering them, or burning a hundred dollars of tokens on a task worth five. Production agent horror stories almost all reduce to a missing guard that costs twenty lines. This is also the week's material customers probe hardest — "what if it does something wrong?" decides deals, and "every action is logged, destructive ones need human approval, and it can only touch these three systems" is the answer that closes them. Everything you build today wraps the D120–122 agent and carries straight into tomorrow's triage checkpoint and Day 137's trajectory evals.

Guided practice

guided 1

Wrap the agent in guards, then watch each one fire

30 min

Create agent_guards.py with the starter code. Guarded wraps your Day 120 TOOLS dict without changing the loop or tools themselves — enforcement lives at the boundary the model cannot cross.
Run scenario 1 (loop): a stuck ScriptedLLM re-reads the same missing file forever. The loop detector trips at 3 identical calls and ends the run with a legible status. Check the audit JSONL: the repetition is visible at a glance.
Run scenario 2 (cost): each call charges the meter; the run stops at the ceiling with partial results — degraded, not silent.
Run scenario 3 (approval): the agent requests delete_file. The gate pauses, prints the (tool, args) for review, and the scripted human DENIES. Confirm the denial arrives as an observation and the agent continues with a non-destructive alternative — steering, not crashing.
Read the audit trail end to end and answer: could you reconstruct each run without the code? If any line leaves you guessing, add the missing field — that is the actual bar.

🐍 python — editable, runs in your browser

# agent_guards.py — guardrails as tool-layer wrappers around the D120 agent.
import json, time

FS = {"settings.yaml": "rate_limit: 120", "notes.txt": "TODO tidy configs"}
def list_files(): return sorted(FS)
def read_file(name):
    if name not in FS: raise FileNotFoundError(name)
    return FS[name]
def delete_file(name): return FS.pop(name) and f"deleted {name}"

TOOL_CLASS = {"list_files": "read", "read_file": "read", "delete_file": "destructive"}
COST = {"read": 0.01, "destructive": 0.02}      # toy per-call cost, stands in for tokens

class Guarded:
    def __init__(self, tools, approver, max_cost=0.10, max_repeats=3):
        self.tools, self.approve = tools, approver
        self.max_cost, self.max_repeats = max_cost, max_repeats
        self.spent, self.recent, self.audit = 0.0, [], []
    def _log(self, **event):
        event["ts"] = round(time.time(), 2)
        self.audit.append(event)
        print("  audit:", json.dumps(event))
    def call(self, tool, args):
        sig = f"{tool}:{json.dumps(args, sort_keys=True)}"
        self.recent.append(sig)
        if self.recent[-self.max_repeats:].count(sig) >= self.max_repeats:
            self._log(event="LOOP_DETECTED", tool=tool, args=args)
            raise RuntimeError("loop detected: same call repeated; run stopped")
        cost = COST[TOOL_CLASS[tool]]
        if self.spent + cost > self.max_cost:
            self._log(event="COST_CEILING", spent=self.spent)
            raise RuntimeError(f"cost ceiling hit at {self.spent:.2f}; partial results stand")
        if TOOL_CLASS[tool] == "destructive":
            ok = self.approve(tool, args)               # human-in-the-loop
            self._log(event="APPROVAL", tool=tool, args=args, approved=ok)
            if not ok:
                return "APPROVAL DENIED: choose a non-destructive alternative"
        self.spent += cost
        try:
            out = self.tools[tool](**args)
            self._log(event="CALL", tool=tool, args=args, ok=True, spent=self.spent)
            return out
        except Exception as e:
            self._log(event="CALL", tool=tool, args=args, ok=False, error=str(e))
            return f"TOOL ERROR: {e}"

def scripted_human(tool, args):
    print(f"  >> APPROVAL REQUESTED: {tool}({args}) — deny (policy: no deletions)")
    return False

g = Guarded({"list_files": list_files, "read_file": read_file,
             "delete_file": delete_file}, approver=scripted_human)

print("--- scenario 1: stuck loop ---")
try:
    for _ in range(5): g.call("read_file", {"name": "ghost.txt"})
except RuntimeError as e: print("STOP:", e)

print("--- scenario 3: destructive needs approval ---")
print("agent sees:", g.call("delete_file", {"name": "notes.txt"}))
print("notes.txt still present:", "notes.txt" in FS)

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Least privilege: build the toolset per task

12 min

Write toolset_for(task_type) returning only what each task needs: investigate → {list_files, read_file}; cleanup → adds delete_file (destructive-classed, so gated anyway); report → {read_file, write_file} with write_file's wrapper validating the path is inside reports/ (an argument allowlist — misuse prevented by validation, not by hoping).
Rerun scenario 3 with the investigate toolset: the delete request now fails at dispatch ("unknown tool") — the strongest guard is the tool that was never mounted. Compare the two failure messages (unmounted vs denied) and note both returned as observations.
For tomorrow's triage agent, draft its toolset on paper: classify_ticket (read), search_docs (read — your capstone retriever), draft_reply (write, schema-validated), escalate (destructive-classed: it emails a human). Justify each classification in a phrase — this paper lands in tonight's project.

On your own

Red-team your own agent

20 min

Attack the guarded agent three ways, then patch what you find. (1) The near-miss loop: alternate two failing calls (A, B, A, B…) — does your exact-signature detector fire? Upgrade it to trip on no-progress: N consecutive calls with error observations. (2) The salami slicer: many cheap calls that each pass the per-call check but blow the total budget — verify the meter catches the accumulation (and fix it if you only checked per-call). (3) The laundered deletion: add a run_shell(cmd) tool classed "read" and watch the agent (or you, roleplaying it) delete via rm — write the two-line moral about tool classification granularity and why catch-all tools defeat every guard.

Hints: (1) keep a window of outcomes, not just signatures; (3) the fix is not a smarter classifier — it is not mounting catch-all tools, or classing them destructive.

Ship before you stop

Harden the Week 18 agent

Ship agent/guards.py in the agent package: the Guarded wrapper (loop/no-progress detection, cost ceiling, approval gates, JSONL audit trail to a file), toolset_for() least-privilege factories, and integration with run_agent via the on_step hook from Day 120. Tests must cover: loop trip, no-progress trip, cost ceiling with partial-results message, approval denial returned as observation, unmounted-tool dispatch failure, and an audit-trail replay test — parse the JSONL from a run and assert the full action sequence is reconstructable. Add RELIABILITY.md: the failure taxonomy table (failure → detector → response) plus your triage-agent toolset classification from guided 2. This hardened agent is exactly what Day 126 deploys.

Rubric — check what you completed (0/6)

All guards live in the tool layer; the loop and tools are unmodifiedSix test scenarios pass, including audit-replay reconstructionDenials and guard trips return as observations or legible statuses, never bare crashesCost guard caps the accumulated total, not just per-call costRELIABILITY.md maps every taxonomy row to an implemented detector and responseCommitted to the practice repo

Common mistakes & misconceptions

Prompting for safety instead of enforcing it. "Never delete files" in the system prompt is advisory; a tool layer without delete, or with a gate, is enforcement. The prompt asks; the wrapper decides.
Detecting only exact-repeat loops. Agents alternate between two failing approaches too — detect no-progress (consecutive errors, no new observations), not just identical signatures.
Gating everything. If every read needs approval, humans rubber-stamp within an hour and the gate is theater. Reserve approvals for destructive/irreversible; let reads flow.
Non-idempotent actions plus retries. An agent that retries create_ticket makes three tickets; idempotency keys and check-before-act are mandatory once loops and retries exist.
Logging only failures. The audit trail must contain every action — the 3am question is "what did it DO?", and reconstruction needs the boring successful steps too.
Catch-all tools (run_shell, eval) classed as safe. One generic tool launders every forbidden action through an innocent-looking call; don't mount what you can't classify.

Knowledge check

Q1. Why is "never delete files" in the system prompt insufficient as a safety control?

Q2. An agent alternates between two different failing tool calls: A, B, A, B… Which detector catches it?

Q3. An approval gate denies a destructive call. The best system behavior is…

Go deeper — curated resources

articleAnthropic — Building Effective Agents (guardrails & human oversight threads) ↗15 min docsOWASP Top 10 for LLM Applications (excessive agency & related risks) ↗25 min articleSimon Willison — prompt injection series (why prompts can't be the guard) ↗20 min

If you have a third hour

Dry-run and plan-approval modes — Beyond per-call gates: have the agent emit its full intended plan (Day 121's table) for approval BEFORE execution — one human review instead of five interrupts. Trade: less friction, staler approval.

Done means

All three guard scenarios run; each trip explained from the audit trail
Unmounted-vs-denied comparison articulated
All three red-team attacks run and patched (or the moral written for #3)
agent/guards.py + RELIABILITY.md committed, six tests green
Quiz ≥ 2/3

How this connects

← Back: Day 44's least-privilege and authorization thinking now applies to a nonhuman caller; Day 46's idempotency became an agent survival trait; Day 120's budget seatbelt grew into a full control structure; Day 124's consent model got its implementation.

Forward →: Tomorrow's triage agent ships inside these guards. Day 132 attacks them (injection), Day 137 evals trajectories from your audit trails, and Day 142 upgrades the JSONL into real distributed traces.

Unlocks: D126 Week 18 Checkpoint: Support-Triage Agent · D132 Guardrails, Injection & AI Security · D158 Reliability & Incident Response