Day 116 · Smell plus card catalog

Hybrid Search

You will be able to

Name the characteristic failure modes of dense retrieval and of lexical retrieval, with examples
Implement BM25 from scratch and explain what each term in the formula rewards
Fuse dense and lexical rankings with reciprocal rank fusion (RRF) and justify rank-based fusion over score mixing
Layer metadata filters, recency, and source weighting onto hybrid retrieval

Today's ~120 minutes

Spaced-rep: due cards + recite HNSW trade-off from memory (D115)	10 min
ELI5 + tech read; trace the BM25 formula term by term	25 min
Guided: BM25 from scratch + RRF fusion	40 min
Practice: query zoo	20 min
Project: HybridRetriever module	20 min
Quiz + flashcards	5 min

Builds on: Day 115 — Embeddings & vector DBs · Day 114 — Chunking strategies

The analogy

🧸 ELI5⚙ Tech

A bloodhound finds things by smell — it doesn't need the exact name, just the scent of the thing. Ask it for "something about taking time off" and it happily leads you to the vacation shelf. But ask it for "form W-4991-B" and the bloodhound is useless: form numbers have no smell. The card catalog is the opposite: type "W-4991-B" and it finds the exact card instantly — but ask it for "time off" when every card says "vacation" and it returns nothing at all.

Dense retrieval (embeddings) is the bloodhound: brilliant at meaning, blind to exact strings — error codes, SKUs, function names, people's names all smell like nothing. Lexical retrieval (BM25, the algorithm behind classic search engines) is the card catalog: exact words or bust. Hybrid search runs BOTH on every query and merges the two ranked lists, so "why am I getting ERR-4092 after taking time off" — half jargon, half meaning — finds both halves. The merge trick, reciprocal rank fusion, is delightfully dumb: ignore the incomparable scores, reward whatever ranks high on either list.

Why this matters on the job

Real user queries are full of unsmellable tokens — ticket numbers, product codes, API names, invoice IDs — and pure-dense RAG systems fail on exactly the queries enterprise users care most about. "Why didn't it find ERR-4092?" is a bug you will personally field as an FDE. Every serious vector DB (Qdrant, Weaviate, Pinecone, pgvector+tsvector) now ships hybrid because pure dense lost this fight in production. Your capstone's retrieval layer (Day 119) is hybrid from v0, and interviewers love asking "when does semantic search fail?" — today you'll have a crisp, example-backed answer.

Guided practice

guided 1

BM25 from scratch, then watch each retriever fail

25 min

Create hybrid_lab.py with the starter code: a compact BM25 class, the synonym-aware dense stand-in from Day 113 (upgraded with a tiny synonym map so it behaves like real embeddings on this corpus), and a shared 6-doc corpus.
Run query A: "what does ERR-4092 mean". Confirm BM25 nails it (#1) and dense flails — the code shares almost no "meaning" tokens with anything.
Run query B: "how much time off can I carry into next year". Confirm dense nails it (the synonym map bridges time off → vacation) and BM25 returns near-zero scores.
Read the BM25 score method against the formula in the tech section: point to the IDF line, the saturation constant k1, and the length-normalization term b. Change k1 to 0 and re-run — term frequency now barely matters. Put it back.
Record both failure cases; the next exercise fuses them.

🐍 python — editable, runs in your browser

import math, re
from collections import Counter

def tok(t): return re.findall(r"[a-z0-9-]+", t.lower())

CORPUS = {
    "hb-01": "Employees accrue 1.5 vacation days per month. Up to 10 unused vacation days roll over to the next year.",
    "hb-02": "Parental leave is 16 weeks fully paid for all new parents.",
    "hb-03": "Purchases over 500 USD require manager pre-approval and a receipt.",
    "prod-1": "Error ERR-4092 means the monthly API quota is exhausted. Upgrade the plan or wait for the quota reset.",
    "prod-2": "The Insights API rate limit is 120 requests per minute per key.",
    "sec-01": "Report a lost laptop to security within 4 hours. Full-disk encryption is mandatory.",
}

class BM25:
    def __init__(self, docs, k1=1.5, b=0.75):
        self.ids = list(docs)
        self.toks = {i: tok(d) for i, d in docs.items()}
        self.N = len(docs)
        self.avgdl = sum(len(t) for t in self.toks.values()) / self.N
        self.df = Counter(w for t in self.toks.values() for w in set(t))
        self.k1, self.b = k1, b
    def idf(self, w):
        return math.log((self.N - self.df[w] + 0.5) / (self.df[w] + 0.5) + 1)
    def score(self, q, i):
        tf, dl, s = Counter(self.toks[i]), len(self.toks[i]), 0.0
        for w in tok(q):
            if tf[w] == 0: continue
            sat = tf[w] * (self.k1 + 1) / (tf[w] + self.k1 * (1 - self.b + self.b * dl / self.avgdl))
            s += self.idf(w) * sat          # rare-word weight x saturated tf
        return s
    def rank(self, q):
        return sorted(self.ids, key=lambda i: -self.score(q, i))

# dense stand-in: bag-of-words + a synonym map simulating what real
# embeddings learn ("time off" lives near "vacation" on the map of meaning)
SYN = {"time": "vacation", "off": "vacation", "pto": "vacation", "carry": "roll",
       "year": "year", "laptop": "laptop", "stolen": "lost"}
def dense_vec(t):
    return Counter(SYN.get(w, w) for w in tok(t))
def cosine(a, b):
    dot = sum(a[w] * b[w] for w in a if w in b)
    na, nb = math.sqrt(sum(v*v for v in a.values())), math.sqrt(sum(v*v for v in b.values()))
    return dot / (na * nb) if na and nb else 0.0
DENSE = {i: dense_vec(d) for i, d in CORPUS.items()}
def dense_rank(q):
    qv = dense_vec(q)
    return sorted(CORPUS, key=lambda i: -cosine(qv, DENSE[i]))

bm25 = BM25(CORPUS)
for q in ["what does ERR-4092 mean",
          "how much time off can I carry into next year"]:
    print("Q:", q)
    print("  bm25 :", [(i, round(bm25.score(q, i), 2)) for i in bm25.rank(q)[:3]])
    print("  dense:", [(i, round(cosine(dense_vec(q), DENSE[i]), 3)) for i in dense_rank(q)[:3]])

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Reciprocal rank fusion — ten lines that fix both

15 min

Append the RRF function below to hybrid_lab.py.
Run both yesterday's failing queries through hybrid. Confirm each now ranks its correct doc #1: RRF trusts whichever system is confident, because the OTHER system's noise lands at deep ranks worth almost nothing (1/(60+5) ≈ 0.015).
Run the mixed query "ERR-4092 while on time off" and inspect the fused list: both prod-1 and hb-01 should surface — neither retriever alone puts both in its top-2.
Change k from 60 to 1 and re-run: small k over-trusts rank #1 of each list; large k flattens everything. See why 60 is the boring, robust default.
Add a third list: docs sorted by a fake updated date, and fuse all three. Fresher docs get a bounded nudge — this is recency weighting without touching stored scores.

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Build the query zoo

20 min

Create query_zoo.md plus code: 9 queries against today's corpus (extend it to ~10 docs first) — 3 where BM25 must win, 3 where dense must win, 3 mixed. For each, record the top-2 from bm25-only, dense-only, and hybrid, and mark which systems got it right. Deliverable: a table plus a 3-sentence conclusion on WHEN hybrid pays.

Constraints: at least one query with a misspelling, one with an exact identifier, one pure paraphrase. Hints: identifiers and rare jargon → lexical; paraphrase/synonyms → dense; misspellings hurt both, which is a preview of query rewriting (Day 117).

Ship before you stop

Hybrid retrieval module for the capstone

Refactor today's lab into retrieval.py, a clean module your capstone imports on Day 119: class HybridRetriever wrapping a BM25 index and your Day 115 VecStore over the SAME chunk IDs, with search(query, k=5, where=None) that (1) applies metadata filters to both retrievers BEFORE ranking, (2) fuses with RRF, (3) returns (id, fused_score, text, metadata) tuples, and (4) exposes explain(query) returning both raw rankings for debugging — the tool you will reach for every time retrieval looks wrong. Include 5 pytest cases from your query zoo.

Rubric — check what you completed (0/6)

Single ingest path keeps BM25 and vector index in lockstep over shared chunk IDsMetadata filter applied pre-ranking in both retrievers, verified by a testRRF fusion with configurable k; default 60explain() shows per-retriever rankings side by sideFive query-zoo tests pass, covering identifier, paraphrase, and mixed queriesCommitted to the practice repo

Common mistakes & misconceptions

Averaging raw BM25 and cosine scores. They live on different scales and distributions — fuse ranks (RRF) or normalize very carefully; never mix raw scores.
Assuming embeddings handle everything and lexical search is legacy. Exact identifiers, fresh jargon, and names have no semantic neighborhood; dense-only RAG fails its most valuable enterprise queries.
Tokenizing away the identifiers ("ERR-4092" split into "err" and "4092"). Your lexical tokenizer must preserve codes, or the card catalog loses its one superpower.
Filtering after fusion. Apply metadata/permission filters inside both retrievers, or fused top-k can be entirely filtered away — returning nothing for valid queries.
Letting the two indexes drift. If chunks are re-ingested into the vector store but not the BM25 index (or vice versa), fusion silently degrades — one ingest path, two indexes, same IDs.
Boosting recency by mutating stored scores at ingest. Bake policy into query-time fusion instead, so changing the boost never requires re-indexing.

Knowledge check

Q1. A user searches "invoice INV-88291 status" and your dense-only RAG returns generic billing docs. Root cause?

Q2. Why does RRF fuse ranks instead of scores?

Q3. In BM25, what does the k1 saturation term prevent?

Go deeper — curated resources

articlePinecone — getting started with hybrid search ↗20 min docsQdrant docs — hybrid queries & sparse vectors ↗20 min articlePinecone Learning Center — BM25 & sparse retrieval ↗15 min

If you have a third hour

Learned sparse retrieval (SPLADE-style) — A neural model that outputs weighted term expansions — lexical precision with learned synonyms, one index instead of two. The best of both, at higher serving cost.

Done means

BM25 implemented; both single-retriever failure modes demonstrated
RRF fixes both failing queries; k sensitivity explored
Query zoo table with 9 queries and a written conclusion
retrieval.py committed with 5 passing tests
Quiz ≥ 2/3

How this connects

← Back: Day 115 built the bloodhound; today you built the card catalog it was missing — with Day 23's hashing (inverted index) and Day 62's information-as-surprise (IDF is literally rarity-as-informativeness) doing the work.

Forward →: Day 117 adds a reranker after fusion, and Day 119 wires HybridRetriever into your capstone v0. On Day 136 you'll measure exactly how much hybrid lifts retrieval recall over either retriever alone.

Unlocks: D117 Reranking & Query Transforms · D119 Week 17 Checkpoint: Capstone Kickoff — Docs-QA v0