Day 136 · Grading the open-book exam

RAG Evaluation

You will be able to

Separate retrieval quality from generation quality and eval each independently
Compute context precision and context recall by hand in runnable Python
Compute faithfulness/groundedness and answer relevance with simple implementations
Diagnose whether a bad answer is a retrieval failure or a generation failure
Build a RAG error taxonomy and route each failure to the right fix

Today's ~120 minutes

Spaced-rep: due cards (judges, golden sets)	10 min
ELI5 + tech read; Ragas metrics docs	20 min
Guided: precision/recall by hand + faithfulness/relevance	45 min
Practice: diagnose ten failing answers	20 min
Project: RAG eval module for the capstone	15 min
Quiz + flashcards	10 min

Builds on: Day 117 — Reranking & precision@k · Day 134 — Golden sets & the eval loop · Day 135 — LLM-as-judge & calibration

The analogy

🧸 ELI5⚙ Tech

An open-book exam can go wrong in two completely different ways, and blaming "the student" hides which. Either the student opened to the wrong pages (retrieval failed — the right passage was never in front of them), or they had the right pages open and still wrote a wrong answer (generation failed — they ignored or misread what was there). If you only look at the final answer, you cannot tell these apart, and you will "fix" the wrong thing for weeks.

So RAG evaluation splits the grade in two. For retrieval you ask: of the passages we pulled, how many were actually relevant (precision), and did we pull all the passages needed to answer (recall)? For generation you ask: is every claim in the answer actually supported by the retrieved passages (faithfulness — no making things up), and does the answer actually address the question that was asked (relevance)? A low answer with high retrieval but low faithfulness is a hallucination problem; a low answer with bad retrieval is an index problem. Different day, different fix — and you only see it because you graded the two halves separately.

Why this matters on the job

Your capstone IS a RAG system, and "it gave a wrong answer sometimes" is not a debuggable statement. When a customer reports a bad answer, the first question a competent FDE asks is "was the right document even retrieved?" — because the fix for a retrieval miss (better chunking, hybrid search, reranking — Days 114–117) is nothing like the fix for a faithful-to-nothing hallucination (prompt grounding, citation enforcement, a smaller more careful model). Teams that measure only end-to-end answer quality thrash; teams that decompose retrieval vs generation fix the actual bottleneck. This is also the Day 140 harness's core, and the metric vocabulary (faithfulness, context precision/recall, answer relevance) is exactly what Ragas and every RAG-eval tool report — today you implement them so they are never magic.

Watch it happen

Grading the open-book exam — retrieval and generation, scored separately

step 1 / 5

	question	metric
retrieval	did we fetch the right pages?	context precision / recall
generation	did we use them honestly?	faithfulness / relevance

One RAG answer looks fine… but four different things could be silently wrong. The grading grid separates them: retrieval quality vs answer quality, each with its own metrics.

🎯 Predict modespeed1×

Guided practice

guided 1

Context precision and recall by hand

20 min

Create rag_eval/retrieval.py from the starter. For 5 golden queries you provide: the set of truly-relevant chunk ids (your labels) and the ranked list your retriever returned.
Implement context precision (relevant∩retrieved / retrieved), context recall (relevant∩retrieved / relevant), and a rank-aware precision that averages precision@k at the ranks where relevant chunks appear.
Run it and read the per-query and mean numbers. Find the query with high precision but low recall (retrieved clean but incomplete) and the one with high recall but low precision (found everything plus junk).
For each failure type, write the fix in a comment: low recall → chunking/search/reranking (Days 114–117); low precision → reranking/filtering/tighter k.
This is pure set arithmetic — no model calls — which is why retrieval eval is cheap and should run on every change.

🐍 python — editable, runs in your browser

def precision(relevant, retrieved):
    r = set(retrieved)
    return len(set(relevant) & r) / max(1, len(r))

def recall(relevant, retrieved):
    return len(set(relevant) & set(retrieved)) / max(1, len(relevant))

def rank_aware_precision(relevant, ranked):
    # average of precision@k taken at each rank a relevant item appears
    rel, hits, precisions = set(relevant), 0, []
    for k, cid in enumerate(ranked, 1):
        if cid in rel:
            hits += 1
            precisions.append(hits / k)
    return sum(precisions) / max(1, len(precisions))

CASES = [   # (query_id, relevant_ids, ranked_retrieved_ids)
    ("q1", ["c3"],        ["c3", "c9", "c1"]),          # perfect-ish
    ("q2", ["c2", "c5"],  ["c2", "c8", "c4"]),          # got c2, missed c5 -> low recall
    ("q3", ["c7"],        ["c1", "c4", "c7", "c9", "c2"]),  # found it but noisy -> low precision
    ("q4", ["c6", "c6b"], ["c6", "c6b"]),               # perfect
    ("q5", ["c0"],        ["c1", "c2", "c3"]),           # total miss -> recall 0
]
ps, rs = [], []
for qid, rel, ret in CASES:
    p, r = precision(rel, ret), recall(rel, ret)
    ps.append(p); rs.append(r)
    print(f"{qid}: precision={p:.2f} recall={r:.2f} "
          f"rank_prec={rank_aware_precision(rel, ret):.2f}")
print(f"MEAN precision={sum(ps)/len(ps):.2f} recall={sum(rs)/len(rs):.2f}")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Faithfulness and answer relevance with a judge

25 min

Create rag_eval/generation.py. Implement faithfulness as claim-decompose-then-verify: an LLM splits the answer into atomic claims; for each claim, a judge decides if the retrieved context ENTAILS it (yes/no). Faithfulness = supported / total claims.
Run it on two answers to the same question over the same context: one grounded, one where you inject an unsupported but plausible sentence. Watch faithfulness drop for the second — you just caught a hallucination numerically.
Implement a simple answer-relevance proxy: ask an LLM to write the question the answer seems to be answering, embed both questions (reuse any embedding model), and report cosine similarity. A faithful answer to the WRONG question scores low here.
Build the 2×2 diagnosis: combine today's retrieval recall with faithfulness and relevance to label each failing golden case as retrieval-miss / hallucination / wrong-question.
Calibrate sanity-check: hand-label faithfulness on 5 cases and confirm the judge agrees (Day 135) — these metrics are only as trustworthy as the judge under them.

🐍 python — editable, runs in your browser

import anthropic
client = anthropic.Anthropic()

def claims(answer):
    r = client.messages.create(model="claude-sonnet-4-5", max_tokens=400,
        messages=[{"role": "user", "content":
            f"Split this answer into atomic factual claims, one per line, no numbering:\n{answer}"}])
    return [c.strip("- ").strip() for c in r.content[0].text.splitlines() if c.strip()]

def entailed(context, claim):
    r = client.messages.create(model="claude-sonnet-4-5", max_tokens=5,
        messages=[{"role": "user", "content":
            f"Context:\n{context}\n\nClaim: {claim}\n\n"
            "Is the claim fully supported by the context? Answer yes or no."}])
    return r.content[0].text.strip().lower().startswith("y")

def faithfulness(context, answer):
    cs = claims(answer)
    supported = sum(entailed(context, c) for c in cs)
    return supported / max(1, len(cs)), cs

CONTEXT = "Refunds are available within 30 days of purchase with a receipt."
grounded = "You can get a refund within 30 days if you have a receipt."
hallucinated = ("You can get a refund within 30 days with a receipt, "
                "and refunds are processed instantly to your bank account.")  # unsupported
for name, ans in [("grounded", grounded), ("hallucinated", hallucinated)]:
    f, cs = faithfulness(CONTEXT, ans)
    print(f"{name}: faithfulness={f:.2f} over {len(cs)} claims")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Diagnose ten failing answers

20 min

Take (or construct) 10 cases where your capstone gave a wrong or weak final answer. For each, compute context recall (was the evidence retrieved?) and faithfulness (were the answer's claims supported?), and assign a diagnosis: RETRIEVAL-MISS (low recall), HALLUCINATION (good recall, low faithfulness), WRONG-QUESTION (faithful but low answer relevance), or FAITHFUL-BUT-CONTEXT-WRONG (the docs themselves are wrong/stale).

Deliver a table (case, recall, faithfulness, relevance, diagnosis, fix-owner) and a tally: which failure class dominates YOUR system? That tally is your prioritized backlog — if 7 of 10 are retrieval misses, spending the week on prompt-grounding is malpractice.

Hints: the point is that "improve the RAG" is not an action. "Context recall is 0.4, so improve retrieval via reranking" is. Let the decomposition pick the fight.

Ship before you stop

RAG eval module for the capstone

Add evals/rag_eval.py to the capstone and wire it to your golden set. It must, per golden query, compute: context precision and recall (you provide relevant-chunk labels for each golden case — extend GUIDELINES.md to cover chunk-relevance labeling), faithfulness of the generated answer against its retrieved context, and an answer-relevance proxy. Output evals/rag-report.md: mean metrics, the per-case diagnosis table, and a short "where is my system weakest — retrieval or generation?" verdict that names the next fix. Reuse your Day 135 calibrated judge for faithfulness and note its agreement number so the metric is trustworthy. This module is a required component of the Day 140 harness.

Rubric — check what you completed (0/6)

Context precision and recall computed from per-query relevant-chunk labelsFaithfulness implemented as claim-decompose + entailment, not a vibes scoreAnswer-relevance proxy present and distinguishes faithful-but-off-topic answersPer-case diagnosis routes each failure to retrieval vs generation vs dataReport names whether retrieval or generation is the current bottleneck, with evidenceUses the calibrated judge and cites its agreement; committed to the capstone

Common mistakes & misconceptions

Evaluating only the final answer. You cannot tell a retrieval miss from a hallucination that way, and you will fix the wrong subsystem for weeks.
Reporting one blended "RAG score". Retrieval and generation have different fixes; a single number hides which is broken. Always decompose.
Faithfulness without claim decomposition. "Rate how grounded this is 1–10" is noisy; splitting into atomic claims and checking each is what makes it a real metric.
Confusing faithfulness with correctness. An answer can be faithful to retrieved-but-wrong documents; catch that as a data/context problem, not a generation one.
Trusting LLM-based RAG metrics uncalibrated. Faithfulness and relevance judges inherit Day 135's biases — validate them against human labels before believing the numbers.
No chunk-relevance labels. Context precision/recall need ground-truth relevant chunks per query; without them you only have generation metrics and half the picture.

Knowledge check

Q1. An answer is wrong. Context recall is 0.9 (evidence was retrieved) but faithfulness is 0.4. The problem is in…

Q2. Context precision vs context recall — which failure means "the evidence needed to answer never entered the prompt"?

Q3. How is faithfulness best computed so it is a real metric rather than a vibe?

Go deeper — curated resources

docsRagas Documentation — metrics (faithfulness, context precision/recall) ↗30 min articleEugene Yan — Task-Specific LLM Evals (RAG section) ↗20 min articlePinecone Learning Center — RAG evaluation ↗20 min

If you have a third hour

Ragas paper and metric definitions ↗ — Study how Ragas formalizes faithfulness (statement extraction + NLI) and context precision (rank-weighted). You built simplified versions today; the library adds robustness, not magic.

Done means

Context precision and recall computed by hand across ≥ 5 queries
Faithfulness via claim decomposition catches an injected hallucination numerically
Ten failing answers diagnosed and tallied by failure class
rag_eval.py + rag-report.md committed to the capstone
Quiz ≥ 2/3

How this connects

← Back: Context precision/recall are Day 75's precision/recall and Day 117's precision@k on the retrieval set; faithfulness uses the Day 135 calibrated judge; the answer-relevance proxy uses Day 92 embeddings; all of it grades the Day 134 golden set over the Day 119 capstone.

Forward →: Day 137 evaluates agents (trajectories, tool-calls) the same decompose-and-measure way; Day 139 puts confidence intervals on these metrics; Day 140 folds retrieval AND answer scoring into the capstone eval harness.

Unlocks: D137 Agent & Task Evals · D140 Week 20 Checkpoint: Capstone Eval Harness · D160 AI System Design