Human Evaluation
- Identify when human judgment is the only valid signal and when automation suffices
- Write an annotation protocol precise enough for consistent labels
- Measure inter-rater agreement (Cohen's kappa) and act on low agreement
- Design side-by-side (pairwise) human comparisons that avoid bias
- Turn user feedback (thumbs, categories, freeform) into durable eval cases
| Spaced-rep: due cards (agent evals, RAG eval) | 10 min |
| ELI5 + tech read; when humans are the only valid judge | 20 min |
| Guided: two-rater agreement study + feedback pipeline | 45 min |
| Practice: design a blinded pairwise eval | 20 min |
| Project: human-eval round + ground truth | 15 min |
| Quiz + flashcards | 10 min |
Builds on: Day 135 — Judge calibration & Cohen's kappa · Day 134 — Golden sets & labeling guidelines · Day 60 — Sampling — a sample represents a population
A food company can measure a soup's salt, temperature, and viscosity with instruments all day — but eventually somebody has to taste it, because "delicious" is not a number any sensor reports. Some qualities of an AI's output are like that: is this explanation genuinely helpful? is this tone right for a grieving customer? is this summary faithful in the way that matters to a lawyer? An LLM judge can approximate these, but the ground truth — the thing you calibrate the judge AGAINST — comes from people.
The catch is that people are noisy instruments too. Ask two tasters and one calls it perfect, the other bland. So human evaluation is not "just ask someone"; it is a small science: give tasters a written protocol so they judge the same way, have several taste the same bowls, and measure how often they agree. If they don't agree, the fault is usually your recipe for judging, not the soup. And every thumbs-down a real user leaves is a taster volunteering data — captured well, it becomes tomorrow's exam question.
Automated metrics and LLM judges are calibrated against human labels (Day 135) — so somewhere a human must provide ground truth, and doing it well is a distinct skill. For subjective, high-stakes, or novel qualities (helpfulness, safety nuance, brand voice, domain correctness a generalist judge can't assess), humans are the gold standard and the source of the labels everything else is measured against. FDEs run human-eval rounds with customers to establish "what good looks like" in writing. And the feedback pipeline — turning production thumbs-downs into eval cases — is the data flywheel (Day 143) that keeps a system improving. Sloppy human eval (vague protocol, one rater, no agreement check) produces ground truth that is quietly wrong, poisoning every metric downstream.
Guided practice
Run a two-rater agreement study
25 min- Take 15 answers from your capstone golden-set run. Write
human_eval/protocol.md: the exact question, a binary HELPFUL-AND-GROUNDED / NOT label, decision rules, and one positive + one negative example. - Rater A is you. Recruit rater B (a friend, or role-play a second pass a day later with only the protocol in front of you — imperfect but instructive). Both label all 15 independently, no discussion.
- Run
human_eval/agreement.py: compute raw agreement and Cohen's kappa (reuse Day 135's function). Print the items you disagreed on. - Read every disagreement. For each, decide: was the protocol ambiguous, or did a rater slip? Revise the protocol to resolve the ambiguity (add a decision rule), then re-label just those items.
- Recompute kappa. Write two sentences: did tightening the protocol raise agreement? The lesson — low agreement is usually a protocol bug, not a people bug.
Feedback-to-eval-case pipeline
20 min- Create
human_eval/feedback.pymodeling a production feedback capture: a function that, on a thumbs-down, records the full context — question, retrieved chunks, answer, a category tag (wrong / unhelpful / unsafe / off-topic), and optional freeform. - Feed it 8 simulated thumbs-down events with varied categories.
- Cluster them by category and print counts — this is the raw material of the data flywheel (Day 143): the dominant category tells you where to invest.
- Convert the three most severe/representative into golden-set cases (question + criteria) appended to your Day 134
golden_set.jsonl. A thumbs-down that does not become a test case is a lesson thrown away. - Note what you must capture at feedback time to make a case reproducible (inputs + retrieved context + version), and add any missing field to the capture schema.
On your own
Design a blinded pairwise human eval
20 minDesign (and, if you have a second person, run) a side-by-side human eval comparing two versions of your capstone's answers — e.g., before vs after a prompt change. Produce human_eval/pairwise-design.md specifying: how you randomize which side is A/B per item (kill position bias), how you blind the rater to which system is which, the exact question ("which answer is more helpful and grounded, or tie?"), how many items and why (sampling — Day 60), and how you will aggregate (win rate with ties, and note you'll put error bars on it tomorrow).
Then, on 10 items (real or simulated raters), collect verdicts and compute the win rate for version B. State whether the sample is big enough to conclude anything — foreshadowing Day 139's honest answer (probably not at n=10).
Hints: the two silent killers are position bias (randomize sides) and unblinding (a rater who knows which is "the new one" roots for it). A pairwise design that ignores either produces confident garbage.
Human-eval round + calibration ground truth for the capstone
Run a real (even if small) human-eval round on the capstone and commit evals/human/: (1) protocol.md — the annotation protocol, precise enough to reproduce; (2) labels from ≥ 2 raters on ≥ 15 overlapping items with a computed Cohen's kappa and a note on how you improved the protocol if kappa was low; (3) the resulting human ground-truth labels — these become the calibration set your Day 135 judge is measured against and the seed of the Day 140 harness's 10 human labels; (4) a feedback-capture schema and ≥ 3 golden cases promoted from simulated thumbs-downs. The deliverable proves you can produce trustworthy ground truth, not just consume metrics.
Common mistakes & misconceptions
- Using one annotator. A single rater's labels have unknown reliability; multiple raters plus a kappa tell you whether the ground truth is trustworthy at all.
- Reporting raw agreement on skewed labels. 90% "helpful" makes near-random raters look aligned; Cohen's kappa is chance-corrected (Day 135/59).
- Blaming raters for low agreement. Divergence usually means the protocol is ambiguous — fix the decision rules and examples, then re-measure.
- Absolute 1–10 human scores. Humans are inconsistent at absolute scales; blinded, position-randomized pairwise comparisons are far more reliable.
- Unblinded or unrandomized comparisons. Raters favor the side they know is "new," and the first-shown option — both silently bias the win rate.
- Collecting thumbs-downs and never mining them. Feedback that does not become clustered failures and new eval cases is a wasted flywheel (Day 143).
Q1. Two annotators label 100 items "safe/unsafe"; 95 are safe. They agree on 93 items. Raw agreement is 93%, but you should also compute…
Q2. Your two raters have kappa 0.3 (poor). The best first response is…
Q3. Why prefer blinded, side-by-side pairwise comparison over asking humans for an absolute 1–10 score?
Go deeper — curated resources
- Chatbot Arena and preference collection — Large-scale blinded pairwise human preference (Elo from A/B votes) is how open leaderboards rank models. Study its randomization and aggregation as the industrial-scale version of today's pairwise design.
- Annotation protocol written and used by ≥ 2 raters on ≥ 15 items
- Cohen's kappa computed; low agreement diagnosed and re-measured after a protocol fix
- Feedback pipeline turns simulated thumbs-downs into ≥ 3 golden cases
- human/ ground-truth committed for judge calibration
- Quiz ≥ 2/3
← Back: This produces the ground truth that calibrates the Day 135 judge, extends Day 134's labeling guidelines, and applies Day 60's sampling and Day 59's base-rate reasoning to human labels.
Forward →: Day 139 puts confidence intervals on the win rates and pass rates you collected here; Day 140 requires 10 human labels to calibrate the harness's judge; Day 143 builds the production feedback flywheel this previews.