Day 117 · The librarian double-checks the pile

Reranking & Query Transforms

You will be able to

Explain bi-encoder vs cross-encoder architectures and why cross-encoders judge relevance better
Justify two-stage retrieval economics: cheap recall over millions, expensive precision over twenty
Apply query rewriting, expansion, and HyDE, and say which failure each one targets
Implement parent-child (small-to-big) retrieval and measure a precision@k improvement

Today's ~120 minutes

Spaced-rep: due cards + write the RRF formula from memory (D116)	10 min
ELI5 + tech read; rerank-flow visualizer	20 min
Guided: two-stage lab + query transforms	45 min
Practice: rerank-depth economics memo	18 min
Project: extend retrieval.py	20 min
Quiz + flashcards	7 min

Builds on: Day 116 — Hybrid search · Day 115 — Embeddings & vector DBs · Day 94 — Attention (Q/K/V)

The analogy

🧸 ELI5⚙ Tech

The library assistant sprints through the stacks and returns with twenty books that "seem about right" — fast, because she only glanced at the spines. Then the senior librarian sits down with your actual question, opens each of the twenty books, reads a page of each WHILE re-reading your question, and hands you the best three in order. Slow per book — which is exactly why she only does it for twenty, never for the whole library.

That's two-stage retrieval. Stage one (your bi-encoder embeddings + BM25) is the sprinter: it compressed every book into a point on the map long before you arrived, so it can't consider your specific question deeply. Stage two, the reranker, is a cross-encoder: it reads query and document TOGETHER, letting every word of one attend to every word of the other, and scores the pair properly. And sometimes the librarian fixes your QUESTION before searching: "the thing where salary keeps coming when you have a baby?" becomes "parental leave pay policy" — query rewriting. Or she imagines what the perfect answering page would look like and searches for THAT — that trick is called HyDE.

Why this matters on the job

Reranking is the highest-leverage single upgrade in most RAG stacks — teams routinely report the largest quality jump per line of code changed, because stage one optimizes recall ("is the answer somewhere in these 20?") while the LLM needs precision ("are the top 3 actually it?"). It also fights lost-in-the-middle (Day 104): reranking puts the best chunk first, where the model actually reads. Query transforms attack the other half of failures — bad queries, not bad indexes. As an FDE, "we added a reranker and rewrote queries" is the before/after demo that wins renewals; expect interview questions on bi- vs cross-encoders verbatim.

Watch it happen

The librarian double-checks the pile — rerank turns recall into precision

step 1 / 5

Query→Fast retrievaltop-20, ~10 ms→Rerankercross-encoder→Top-3to the prompt→Generate

✉ 20 candidates: the answer is in there… somewhere around rank 7

Fast retrieval (ANN + BM25) is optimized for RECALL: cast a wide net, top-20 candidates in milliseconds. But the net catches plausible-looking junk too.

🎯 Predict modespeed1×

Guided practice

guided 1

Two-stage retrieval with a measurable precision@3 gain

25 min

Create rerank_lab.py with the starter code. Stage one is Day 116's hybrid retriever over a 12-chunk corpus with several near-miss distractors. Stage two is a simulated cross-encoder: it scores (query, doc) PAIRS using phrase-level evidence — bigram matches and coverage of query terms — which a bag-of-words bi-encoder structurally cannot see. (In local runs you'd swap in sentence-transformers' CrossEncoder — the harness is identical.)
Run the labeled eval: 5 queries, each with known relevant chunk IDs. The script prints precision@3 for stage-one-only vs reranked.
Confirm the mechanism on query 1: stage one ranks the distractor ("vacation REQUEST process") above the answer ("vacation ACCRUAL policy") because single words overlap; the pair-scorer sees the bigram "roll over" and flips them.
Record your precision@3 lift. Then set RERANK_DEPTH to 3 (rerank only stage one's top-3) and watch the gain shrink: a reranker cannot promote what stage one never fetched. Recall first, then precision.

🐍 python — editable, runs in your browser

import math, re
from collections import Counter

CHUNKS = {
    "hr-1": "Vacation accrual policy: 1.5 days per month accrue; up to 10 unused days roll over into next year.",
    "hr-2": "Vacation request process: submit requests in Workday at least 2 weeks ahead.",
    "hr-3": "Sick leave: unlimited with manager notification; doctor note after 5 consecutive days.",
    "hr-4": "Parental leave is 16 weeks fully paid, plus 4-week flexible return.",
    "fin-1": "Purchases over 500 USD require manager pre-approval before purchase.",
    "fin-2": "Expense reports are due within 30 days; late reports need VP sign-off.",
    "fin-3": "Corporate card holders must reconcile statements monthly.",
    "it-1": "Password policy: 14+ characters, rotated yearly, MFA everywhere.",
    "it-2": "Lost laptops: report to security within 4 hours; remote wipe follows.",
    "api-1": "ERR-4092 means monthly API quota exhausted; upgrade plan or wait for reset.",
    "api-2": "Insights API rate limit is 120 requests per minute per key.",
    "api-3": "API keys rotate every 90 days via the developer console.",
}
# labeled eval set: query -> relevant chunk ids
EVAL = [
    ("how many unused vacation days roll over", {"hr-1"}),
    ("do I need approval before buying a 700 dollar monitor", {"fin-1"}),
    ("stolen laptop what do I do", {"it-2"}),
    ("why does the api say quota exhausted", {"api-1"}),
    ("deadline for submitting expenses", {"fin-2"}),
]

def tok(t): return re.findall(r"[a-z0-9-]+", t.lower())
def vec(t): return Counter(tok(t))
def cosine(a, b):
    dot = sum(a[w] * b[w] for w in a if w in b)
    na, nb = math.sqrt(sum(v*v for v in a.values())), math.sqrt(sum(v*v for v in b.values()))
    return dot / (na * nb) if na and nb else 0.0
VECS = {i: vec(c) for i, c in CHUNKS.items()}

def stage1(q, k=8):                      # bi-encoder-style: independent vectors
    qv = vec(q)
    return sorted(CHUNKS, key=lambda i: -cosine(qv, VECS[i]))[:k]

def cross_score(q, doc):
    """Simulated cross-encoder: judges the PAIR. Sees bigram evidence and
    query-term coverage — interactions invisible to independent embeddings.
    Locally: from sentence_transformers import CrossEncoder;
    CrossEncoder('cross-encoder/ms-marco-MiniLM-L6-v2').predict([(q, doc)])"""
    qt, dt = tok(q), tok(doc)
    dset = set(dt)
    coverage = sum(1 for w in set(qt) if w in dset) / max(len(set(qt)), 1)
    qbi = {(a, b) for a, b in zip(qt, qt[1:])}
    dbi = {(a, b) for a, b in zip(dt, dt[1:])}
    bigram = len(qbi & dbi)
    return 2.0 * bigram + coverage

RERANK_DEPTH = 8
def two_stage(q, k=3):
    cands = stage1(q, k=RERANK_DEPTH)
    return sorted(cands, key=lambda i: -cross_score(q, CHUNKS[i]))[:k]

def precision_at_k(ranked, relevant, k=3):
    return sum(1 for i in ranked[:k] if i in relevant) / k

p1 = p2 = 0
for q, rel in EVAL:
    s1, s2 = stage1(q, k=3), two_stage(q, k=3)
    p1 += precision_at_k(s1, rel); p2 += precision_at_k(s2, rel)
    print(f"Q: {q}\n   stage1: {s1}   reranked: {s2}   relevant: {sorted(rel)}")
print(f"\nprecision@3  stage1-only: {p1/len(EVAL):.2f}   reranked: {p2/len(EVAL):.2f}")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Query transforms: rewrite, HyDE, small-to-big

20 min

Append the starter code. It demonstrates three transforms against the same corpus.
Rewriting: the conversational query "and what about carrying them over?" retrieves garbage alone. The rewritten, self-contained version (simulating what an LLM produces given chat history) retrieves hr-1. In production the rewrite prompt is: "Given the conversation, rewrite the last user message as a fully self-contained search query."
HyDE: the query "use it or lose it?" shares no vocabulary with any chunk. The fake hypothetical answer (write it yourself: one sentence a policy doc WOULD say) retrieves correctly — wrong facts, right vocabulary, right neighborhood. Verify by making your hypothetical say "20 days roll over" (factually wrong) and watching retrieval still succeed.
Small-to-big: sentences are indexed as children pointing at parent sections; retrieval matches a crisp sentence, but the returned context is the whole parent. Confirm the returned parent contains BOTH the accrual rate and the rollover cap — completeness the child alone lacked.

🐍 python — editable, runs in your browser

# --- 1. query rewriting (multi-turn) ---
history = ["user: how do vacation days accrue?",
           "assistant: 1.5 days per month.",
           "user: and what about carrying them over?"]
raw_q = "and what about carrying them over?"
rewritten = "how many unused vacation days carry over to next year"  # LLM output
print("raw:      ", stage1(raw_q, k=2))
print("rewritten:", stage1(rewritten, k=2))

# --- 2. HyDE: embed a hypothetical ANSWER, not the question ---
q = "use it or lose it?"
hyde_doc = ("Unused vacation days roll over to the next year up to a cap; "
            "days beyond the cap are forfeited.")            # LLM-imagined answer
print("query-embed:", stage1(q, k=2))
print("hyde-embed: ", stage1(hyde_doc, k=2))

# --- 3. small-to-big: index children, deliver parents ---
PARENTS = {
    "hr-A": "Vacation: employees accrue 1.5 days per month. Up to 10 unused days roll over into next year. Days beyond 10 are forfeited on Jan 1.",
    "fin-A": "Expenses: purchases over 500 USD need pre-approval. Reports due within 30 days. Late reports need VP sign-off.",
}
CHILDREN = {}   # child id -> (sentence, parent id)
for pid, ptext in PARENTS.items():
    for j, sent in enumerate(s.strip() for s in ptext.split(". ") if s.strip()):
        CHILDREN[f"{pid}#c{j}"] = (sent, pid)

def small_to_big(q, k=1):
    qv = vec(q)
    ranked = sorted(CHILDREN, key=lambda c: -cosine(qv, vec(CHILDREN[c][0])))
    out, seen = [], set()
    for cid in ranked:                       # dedupe children of same parent
        pid = CHILDREN[cid][1]
        if pid not in seen:
            seen.add(pid); out.append((cid, pid, PARENTS[pid]))
        if len(out) == k: break
    return out

for cid, pid, ptext in small_to_big("what happens to days beyond the cap"):
    print(f"matched child {cid} -> deliver parent {pid}:\n   {ptext}")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Rerank-depth economics memo

18 min

Your capstone will rerank with a cross-encoder at ~30 ms per (query, doc) pair, after a stage one that takes ~50 ms total. Write a short memo (code optional) answering: (1) total added latency at rerank depths 5, 20, 50, 100 assuming sequential scoring, then assuming batches of 16 scored in parallel at 40 ms/batch; (2) using your lab result that gains shrink with depth, which depth do you recommend for an interactive docs-QA UI with a 2-second answer budget, and why; (3) one situation where you would rerank with an LLM prompt instead of a cross-encoder.

Hints: depth × 30 ms sequential; ceil(depth/16) × 40 ms batched. LLM-as-reranker fits when volume is tiny and you already pay for the LLM call.

Ship before you stop

Add rerank + transforms to the capstone retriever

Extend Day 116's retrieval.py: (1) a Reranker class with the pair-scoring interface (score(query, text) -> float), shipping today's heuristic scorer and a documented drop-in stub for sentence-transformers' CrossEncoder locally; (2) HybridRetriever.search(..., rerank_depth=20) running fuse-then-rerank; (3) a rewrite_query(history, query) hook that returns the prompt to send an LLM (string in, string out — testable without an API key); (4) the labeled 5-query eval from the lab as a pytest that asserts reranked precision@3 ≥ stage-one precision@3. Commit with before/after numbers in the README.

Rubric — check what you completed (0/6)

Reranker interface identical for heuristic and CrossEncoder backendsrerank_depth configurable; fuse-then-rerank order verified by a testQuery-rewrite hook produces a self-contained query prompt from historyEval test passes: reranked precision@3 ≥ stage-one baselineREADME records the measured precision@3 liftCommitted to the practice repo

Common mistakes & misconceptions

Reranking to fix a recall problem. If the answer isn't in stage one's candidates, no reranker can surface it — widen k, fix chunking, or add hybrid FIRST; rerankers only reorder.
Using a cross-encoder as the primary retriever. One forward pass per document per query does not scale past thousands of docs — that's the whole reason two stages exist.
Confusing the two encoder types: bi-encoders embed separately (indexable, weaker); cross-encoders read the pair jointly (unindexable, stronger). Interviews ask exactly this.
Skipping query rewriting in multi-turn chat. "What about the second one?" embeds to mush; conversational RAG without rewrite fails on most follow-ups.
Expecting HyDE to add knowledge. It only fixes vocabulary mismatch by moving the query into document space; the hypothetical's wrong facts are never shown to the user.
Measuring reranker value by vibes. Precision@k on a labeled query set is 20 lines of code (you wrote them today); without it you cannot justify the added latency.

Knowledge check

Q1. Why can a cross-encoder outrank a bi-encoder on the same (query, document) pair?

Q2. Retrieval recall@20 is 55% — the right chunk is usually absent from the top 20. Adding a reranker will…

Q3. What makes HyDE work despite the hypothetical document containing wrong facts?

Go deeper — curated resources

articlePinecone — rerankers & two-stage retrieval ↗25 min paperHyDE paper — Precise Zero-Shot Dense Retrieval (Gao et al.) ↗25 min docsSentence-Transformers — cross-encoder usage ↗15 min

If you have a third hour

Lost-in-the-middle mitigation by ordering — After reranking, place the best chunk first and second-best last (ends of the context) rather than strictly in order — models attend to edges. Try it on Day 119.

Done means

Precision@3 lift measured and the depth experiment explained
All three transforms demonstrated, including the wrong-facts HyDE test
Economics memo written with latency math
retrieval.py extended; eval test green; lift recorded in README
Quiz ≥ 2/3

How this connects

← Back: Day 94's attention is literally why cross-encoders win — joint attention over the pair. Day 116's RRF reappears as the fusion step before reranking, and Day 114's size dilemma just got its structural fix in small-to-big.

Forward →: Day 118 chains these transforms into multi-hop and agentic retrieval. Your capstone v0 (Day 119) ships fuse-then-rerank, and Day 136 upgrades today's 5-query eval into a 40-case Ragas-style harness.

Unlocks: D118 Advanced RAG Patterns · D119 Week 17 Checkpoint: Capstone Kickoff — Docs-QA v0 · D136 RAG Evaluation