Day 140 · The exam is now automated

Week 20 Checkpoint: Capstone Eval Harness

You will be able to

Recall the Week 20 eval toolkit (golden sets, judges, RAG/agent metrics, statistics) from memory
Assemble a single automated harness that scores retrieval AND answers on the capstone
Calibrate the harness's judge against 10 human labels and report agreement
Produce an eval report with confidence intervals and an honest bottleneck verdict
Wire the harness so it is ready to become the Day 141 CI regression gate

Today's ~115 minutes

Spaced-rep: due Week 20 deck (all six lessons)	10 min
Closed-book recall of Week 20 + diff against notes	20 min
Assemble and run the unified harness end-to-end	40 min
Practice: calibrate the judge on 10 human labels	20 min
Project: finalize harness + eval report	15 min
Cumulative quiz + write gap flashcards	10 min

Builds on: Day 134 — Golden sets & the eval loop · Day 136 — RAG evaluation metrics · Day 139 — Statistics & confidence intervals for evals

The analogy

🧸 ELI5⚙ Tech

All week you forged individual instruments: a golden set (the exam), a calibrated judge (a fair grader), retrieval and generation metrics (separate grades for finding vs writing), agent scorers, and statistics (error bars). Today you bolt them into one machine and press a button. You feed in your Docs-QA capstone; out comes a report card that grades the whole system the way a real examiner would — and does it again, identically, every time you change anything.

This is the payoff of the whole phase. Before this week, "is it good?" was answered by chatting with it and feeling optimistic. Now it is answered by a number, with error bars, broken down into "retrieval is the weak part, not generation," calibrated against real human judgment. The exam that used to require you to sit and grade by hand now runs while you get coffee — and tomorrow you'll rig it to a tripwire so a bad change can't even reach production. A review day, so first you rebuild the week's ideas from a blank page: if you can't reconstruct the eval-driven loop and what kappa protects against, you learned recognition, not the skill.

Why this matters on the job

A capstone with an automated eval harness is a fundamentally different portfolio piece than one without — it is the single clearest signal that you build like a 2026 AI engineer rather than a demo-hacker. In interviews, "walk me through how you evaluate your system" is answered by showing this harness. For an FDE, it is the artifact that lets you tell a customer "here is our quality number, here is its uncertainty, here is where it's weak, and here is the gate that keeps it from regressing." Everything after this — CI gates (D141), tracing (D142), drift (D145), the hardening finale (D177) — assumes this harness exists.

Watch it happen

The exam written before the student exists — eval-driven development

step 1 / 5

Golden set40+ cases→Run appall cases→Gradecode + judge→Reportscore + CIs→Changeprompt/retrieval

✉ {"q": "leave policy for contractors?", "must_cite": "HR §3.2"}

Before improving anything, freeze a golden set: real questions with verified answers. This is your exam.

🎯 Predict modespeed1×

Guided practice

guided 1

Closed-book recall of Week 20

15 min

Close all notes. On a blank page, reproduce:

The eval-driven development loop, in one line.
The four golden-set intents and why negative/refusal cases are non-negotiable.
The three LLM-judge biases and one mitigation each.
Why you report Cohen's kappa, not just raw agreement.
Context precision vs recall, and the retrieval-vs-generation 2×2 failure diagnosis.
Outcome vs trajectory scoring, and why tools get mocked.
Why a pass rate needs a confidence interval, and why A/B should be paired.

Diff against Days 134–139. Score each; every miss becomes a flashcard written today. This diff, not the re-reading, is the point of a review day — recognition masquerades as mastery until you face a blank page.

guided 2

Assemble and run the unified harness

35 min

Create evals/harness.py that imports your week's pieces: golden-set loader (D134), retrieval scorer (D136), programmatic + judge answer scorer (D134/D135/D136), optional agent scorer (D137), and the bootstrap CI (D139).
Make it callable headless: python -m evals.harness --set evals/golden_set.jsonl prints a JSON summary — overall pass rate with CI, mean context precision/recall with CIs, mean faithfulness, judge agreement, and the list of failing case ids with their diagnosis (retrieval vs generation).
Ensure ≥ 40 golden cases (extend from Day 134; fold in Day 133 red-team cases and Day 138 feedback cases). Confirm coverage across intents.
Run it end-to-end on your capstone. Read the bottleneck verdict: is retrieval or generation your weak point? That single sentence drives your next week of work.
Save the JSON summary to evals/latest.json — this is the artifact the Day 141 gate will compare against.

🐍 python — editable, runs in your browser

import json, random, argparse

def bootstrap_ci(xs, B=4000, alpha=0.05):
    n = len(xs); m = []
    for _ in range(B):
        m.append(sum(xs[random.randrange(n)] for _ in range(n)) / n)
    m.sort()
    return sum(xs)/n, m[int(alpha/2*B)], m[int((1-alpha/2)*B)]

def run_harness(golden_path, answer_fn, retrieve_scores, judge_fn):
    cases = [json.loads(l) for l in open(golden_path)]
    passes, cprec, crec, faith, fails = [], [], [], [], []
    for c in cases:
        ans = answer_fn(c["question"])
        ok = judge_fn(c, ans)                      # calibrated judge / programmatic
        passes.append(int(ok))
        rp, rr = retrieve_scores(c)                # context precision/recall
        cprec.append(rp); crec.append(rr)
        if not ok: fails.append(c["id"])
    p, lo, hi = bootstrap_ci(passes)
    return {"n": len(cases),
            "pass_rate": round(p, 3), "pass_ci": [round(lo,3), round(hi,3)],
            "context_precision": round(sum(cprec)/len(cprec), 3),
            "context_recall": round(sum(crec)/len(crec), 3),
            "failing": fails,
            "bottleneck": "retrieval" if sum(crec)/len(crec) < 0.7 else "generation"}

if __name__ == "__main__":
    ap = argparse.ArgumentParser(); ap.add_argument("--set", default="evals/golden_set.jsonl")
    args = ap.parse_args()
    # summary = run_harness(args.set, answer, retrieve_scores, judge)
    # json.dump(summary, open("evals/latest.json", "w"), indent=2)
    # print(json.dumps(summary, indent=2))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Calibrate the harness judge on 10 human labels

20 min

Hand-label 10 golden cases PASS/FAIL yourself (your ground truth — the Day 138 protocol applies). Run the harness's judge on the same 10, compute raw agreement and Cohen's kappa (Day 135), and record them in evals/harness-calibration.md.

If kappa is below ~0.6, iterate the judge prompt or tighten the golden-set criteria until it clears — or, honestly, document that this judge is only trustworthy on part of the distribution and mark which metrics rely on it. Then state, in one sentence per judged metric, whether you trust it given the calibration.

Hints: 10 labels is the checkpoint minimum, not a comfortable number — the CI on kappa from 10 items is wide (Day 139!). Note that too: your calibration itself has uncertainty. The point is the discipline, extended as the set grows.

Ship before you stop

Capstone Eval Harness v1 (Week 20 checkpoint)

Ship the Week 20 capstone gate. Commit to the capstone repo: (1) evals/golden_set.jsonl with ≥ 40 cases covering all topics, all four intents, ≥ 5 negatives, plus red-team (D133) and feedback (D138) cases; (2) evals/harness.py, invocable headless, scoring BOTH retrieval (context precision/recall) AND answers (programmatic + calibrated judge, faithfulness), emitting JSON with confidence intervals; (3) evals/harness-calibration.md — judge calibrated on ≥ 10 human labels with raw agreement + kappa; (4) evals/EVAL-REPORT.md — every metric with a 95% CI, per-case failure diagnosis, the retrieval-vs-generation bottleneck verdict, and a proposed regression threshold (outside the noise band) ready for Day 141. Tag it as the eval milestone. This is a headline artifact of your portfolio.

Rubric — check what you completed (0/6)

Golden set ≥ 40 cases, all four intents, ≥ 5 negatives, incl. red-team + feedback casesHarness runs headless and scores retrieval AND answers in one invocationJudge calibrated on ≥ 10 human labels; raw agreement AND kappa reportedEVAL-REPORT.md gives every metric a 95% CI and names the current bottleneck with evidenceA regression threshold outside the noise band is proposed for the Day 141 gateClosed-book Week 20 recall diffed against notes; gap flashcards written

Common mistakes & misconceptions

Shipping a harness that only scores final answers. Without retrieval metrics you cannot tell a retrieval miss from a hallucination — the whole Day 136 point.
An uncalibrated judge in the harness. A judge with unknown agreement makes every judged number untrustworthy; calibrate on human labels and report kappa.
Reporting metrics without CIs. On a 40-case set the noise band is wide; bare numbers invite shipping decisions on jitter (Day 139).
A harness you can only run by hand in a notebook. If it is not headless and machine-readable, it cannot become tomorrow's CI gate — build for D141 now.
A golden set missing negatives and red-team cases. It reports a flattering number and lets injection/refusal regressions through silently.
Treating the review day as re-reading. The closed-book diff is where gaps surface; recognition feels like mastery and is not.

Knowledge check

Q1. The Day 140 harness reports pass rate 79% (95% CI 67–89%), context recall 0.55, faithfulness 0.94. Where is the system's bottleneck?

Q2. Why must the harness be invocable headless with a machine-readable summary?

Q3. From memory: your judge agrees with 10 human labels 90% of the time but Cohen's kappa is 0.1. What do you do?

Go deeper — curated resources

articleHamel Husain — Your AI Product Needs Evals ↗25 min docsRagas Documentation — assembling a RAG eval ↗20 min docspromptfoo Documentation — running eval suites ↗20 min articleEugene Yan — Task-Specific LLM Evals ↗15 min

If you have a third hour

promptfoo and CI-ready eval harnesses ↗ — Study how promptfoo defines a headless, config-driven eval that returns pass/fail for CI. Your hand-built harness has the same shape — knowing the tooling helps you scale it after Demo Day.

Done means

Closed-book Week 20 recall completed and diffed; gap flashcards written
Unified harness runs headless and scores retrieval AND answers with CIs
Judge calibrated on ≥ 10 human labels; agreement + kappa recorded
EVAL-REPORT.md names the bottleneck and proposes a noise-band regression threshold
Golden set ≥ 40 cases committed with red-team + feedback cases; cumulative quiz ≥ 2/3

How this connects

← Back: This fuses the whole week: golden sets (D134), judges (D135), RAG metrics (D136), agent scoring (D137), human ground truth (D138), and statistics (D139) — over the Day 119 capstone, hardened by the Day 133 red-team.

Forward →: Tomorrow (Day 141) this harness becomes a CI regression gate using the noise-band threshold you set. Day 142 traces the requests it scores, Day 145 watches these metrics drift in production, and Day 177 runs this harness as a release gate before Demo Day.

Unlocks: D141 Regression Gates & CI for AI · D142 Tracing LLM Applications · D177 Capstone Quality Gates