Day 135 · Who grades the grader?

Graders — Code, Rubric & LLM-as-Judge

You will be able to

Match grader type (exact, programmatic, rubric, LLM-judge) to output type
Build an LLM-as-judge with a clear rubric and structured verdict
Demonstrate judge biases in code: position, verbosity, and self-preference
Calibrate a judge against human labels and report its agreement rate
Choose between pairwise and pointwise judging and know each one's failure modes

Today's ~120 minutes

Spaced-rep: due cards (golden sets, Week 19)	10 min
ELI5 + tech read; Hamel LLM-judge guide	20 min
Guided: demonstrate biases + calibrate a judge	45 min
Practice: harden the judge prompt	20 min
Project: calibrated judge for the capstone	15 min
Quiz + flashcards	10 min

Builds on: Day 110 — Structured outputs (judge verdicts) · Day 134 — Golden sets & the eval loop · Day 59 — Bayes/base rates — reading agreement honestly

The analogy

🧸 ELI5⚙ Tech

Yesterday you graded with substring checks — cheap and reliable, but blind. "Must contain 30 days" passes "the refund window is definitely not 30 days." For open-ended answers you need a grader that reads for meaning, so people reach for a second AI: let one model judge another's answers against a rubric. Powerful — and quietly biased in ways a human grader isn't.

Three biases you can measure today. Position: show a judge answer A then answer B and it favors whichever came first, even when they're swapped. Verbosity: it rewards the longer, more confident-sounding answer, even when the short one is right. Self-preference: it prefers answers written in its own style. So "who grades the grader?" isn't a riddle — it's a required step. Before you trust a judge's numbers, you check them against a human's on the same cases and measure how often they agree. A judge you haven't calibrated is a bathroom scale you've never checked: the number is precise, confident, and possibly off by fifteen pounds.

Why this matters on the job

LLM-as-judge is how modern teams score open-ended output at scale — nobody hand-grades 40,000 answers. But an uncalibrated judge produces authoritative-looking numbers that are wrong, and shipping decisions ride on them. The senior move is to treat the judge as a system that itself needs evaluating: build it, measure its agreement with humans, and report that agreement alongside its scores. Interviewers probe exactly this ("how do you know your judge is any good?"), and for an FDE, a calibrated judge is what lets you promise a customer a quality number you can actually defend.

Watch it happen

Who grades the grader? — LLM-as-judge, biased then fixed

step 1 / 5

answer A (correct)

"Refunds within 30 days with receipt."

answer B (wrong)

"Great question! Our generous policy…

…refunds within 90 days, no receipt…"

Two answers to the same support question. A judge model must pick the better one. Answer A is correct and concise; answer B is longer, prettier — and factually wrong about the refund window.

🎯 Predict modespeed1×

Guided practice

guided 1

Demonstrate position and verbosity bias in code

25 min

Create judge_bias.py. It defines a pairwise judge (asks a model which of answer A / answer B better answers a question) returning "A" or "B".
Position bias: take 8 (question, good_answer, mediocre_answer) triples. Judge each TWICE — once with the good answer as A, once as B. Count how often the verdict flips purely from swapping order. Any flip is pure position bias; print the flip rate.
Verbosity bias: for 6 questions, build a SHORT correct answer and a LONG answer that is correct-but-padded (or subtly worse but wordy). Judge them. Count how often length wins over correctness.
Mitigation: implement "both-orders" judging — only declare a winner if it wins in both positions, else TIE. Re-run the position test and show the flip-driven wins collapse to ties.
Record all three rates. Write two sentences: which bias was strongest for your model, and why reporting an uncalibrated pairwise number would have misled you.

🐍 python — editable, runs in your browser

import anthropic
client = anthropic.Anthropic()

def judge_pairwise(question, a, b):
    prompt = (f"Question: {question}\n\nAnswer A: {a}\n\nAnswer B: {b}\n\n"
              "Which answer is better? Reply with exactly one character: A or B.")
    r = client.messages.create(model="claude-sonnet-4-5", max_tokens=5,
                               messages=[{"role": "user", "content": prompt}])
    return r.content[0].text.strip()[:1].upper()

def judge_both_orders(question, good, bad):
    v1 = judge_pairwise(question, good, bad)   # good is A
    v2 = judge_pairwise(question, bad, good)   # good is B
    if v1 == "A" and v2 == "B": return "GOOD"  # consistent
    if v1 == "B" and v2 == "A": return "BAD"   # consistent (judge prefers bad)
    return "TIE"                                # order-dependent -> untrustworthy

PAIRS = [("What is the refund window?",
          "Refunds within 30 days with a receipt.",           # good, short
          "Our company deeply values customers and strives to ensure "
          "satisfaction through a variety of flexible policies.")]  # padded, empty
flips = 0
for q, good, bad in PAIRS * 8:
    a = judge_pairwise(q, good, bad); b = judge_pairwise(q, bad, good)
    # position bias: 'good' should win regardless of slot
    if not (a == "A" and b == "B"): flips += 1
print("order-inconsistent verdicts:", flips, "/", len(PAIRS) * 8)
print("both-orders verdict:", judge_both_orders(*PAIRS[0]))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Calibrate a pointwise judge against human labels

20 min

Create judge_calibrate.py. Take 20 answers from your Day 134 golden-set run and hand-label each PASS/FAIL yourself (this is your human ground truth — do it before looking at the judge).
Build a pointwise judge: given the question, the answer, and your must-include criteria, it returns a structured verdict {verdict: PASS|FAIL, reason}. Pin model + temperature 0.
Run it on the 20 cases. Build the confusion matrix vs your labels.
Compute raw agreement AND Cohen's kappa with the from-scratch code in the starter. Notice: if your labels are 17 PASS / 3 FAIL, a judge that always says PASS gets 85% raw agreement but kappa ~0 — the Day 59 base-rate trap, in an eval.
Decide: is this judge trustworthy on this distribution? State the agreement and kappa as a property you would report alongside any judge-scored eval.

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Harden a judge prompt against its biases

20 min

Take your pointwise judge and iterate its prompt to raise agreement/kappa with your human labels on the 20-case set. Try, one at a time: (1) replacing a 1–5 scale with binary PASS/FAIL + a required one-line critique; (2) adding the explicit must-include/must-not criteria into the judge prompt so it grades against your definition, not its taste; (3) instructing it to ignore answer length and style and judge only grounding + correctness.

Report a small table: judge-prompt version × (raw agreement, kappa). Keep the version that best matches humans WITHOUT overfitting to these 20 cases (hold out 5 to check). End with one sentence on the residual disagreements — are they judge errors, or cases where YOUR labels were actually inconsistent (a Day 138 signal)?

Hints: sometimes calibration reveals the human labels were the noisy ones. When the judge and you disagree, read the case — occasionally the judge is right and your guideline was vague.

Ship before you stop

Calibrated judge for the capstone eval

Add evals/judge.py and evals/judge-calibration.md to the capstone repo. Build a pointwise LLM-as-judge that scores answers against your golden-set criteria and returns a structured verdict. Calibrate it: hand-label ≥ 15 cases (grow toward the 10-human-label bar Day 140 requires), run the judge, and report raw agreement and Cohen's kappa in the calibration doc, with the confusion matrix. Document the judge model, temperature, and prompt version (it is part of your measurement apparatus). State plainly the distribution on which the judge is validated and where you would NOT trust it. Include your position/verbosity-bias findings as a short appendix so a reviewer knows you tested for them.

Rubric — check what you completed (0/6)

Judge returns a structured, criterion-referenced verdict — not a bare 1–10Calibrated against ≥ 15 hand-written human labels with a confusion matrixReports BOTH raw agreement and Cohen's kappa, and interprets the kappa honestlyJudge model, temperature, and prompt version pinned and documentedBias appendix shows position/verbosity were tested with resultsjudge.py and judge-calibration.md committed to the capstone

Common mistakes & misconceptions

Trusting judge scores without calibration. An uncalibrated judge gives precise, confident, possibly-wrong numbers; always report agreement/kappa with humans first.
Reading raw agreement alone on skewed classes. 90% PASS labels make "always PASS" look 90% accurate; Cohen's kappa exposes the uselessness (Day 59 base rates).
Ignoring position bias in pairwise judging. First/last slot wins disproportionately; run both orders and require a consistent winner or call it a tie.
Rewarding verbosity. Judges favor long, assertive answers; rubrics must reward grounding and correctness explicitly and control for length.
Using the same model to write and judge answers. Self-preference inflates scores; use a different judge model where feasible.
Fine-grained scales (1–10) as the primary signal. They add noise and agree with humans worse than binary pass/fail plus a critique.

Knowledge check

Q1. A pairwise judge picks answer A 80% of the time. When you swap the answers into the other order, it still picks the first-shown answer 78% of the time. What is happening?

Q2. Your judge agrees with human labels 88% of the time, but Cohen's kappa is 0.05. What does that tell you?

Q3. For "did prompt version B beat version A overall?", which grader design is generally most reliable?

Go deeper — curated resources

articleHamel Husain — LLM-as-a-Judge: A Complete Guide ↗30 min paperJudging LLM-as-a-Judge (MT-Bench) — Zheng et al. ↗30 min articleEugene Yan — Task-Specific LLM Evals ↗20 min

If you have a third hour

Critique shadowing (Hamel) ↗ — Have a domain expert write pass/fail + critique first, then build the judge to match those critiques iteratively. The critique, not the score, is where the alignment happens.

Done means

Position and verbosity bias demonstrated with measured rates
Both-orders mitigation implemented and shown to reduce order-driven flips
Judge calibrated vs ≥ 15 human labels; raw agreement AND kappa reported
judge.py + calibration doc committed to the capstone
Quiz ≥ 2/3

How this connects

← Back: The judge emits Day 110 structured verdicts; kappa on skewed classes is Day 59's base-rate trap; the judge grades the golden set you built on Day 134.

Forward →: Day 136 uses judges to score RAG faithfulness and relevance; Day 138 formalizes human labels and inter-rater agreement; Day 139 puts error bars on judge-scored pass rates; Day 140 requires a judge calibrated on 10 human labels in the harness.

Unlocks: D136 RAG Evaluation · D138 Human Evaluation · D141 Regression Gates & CI for AI