Day 113 · The open-book exam

RAG I — Architecture

You will be able to

Draw the full RAG pipeline (ingest → chunk → embed → index → retrieve → generate) and explain what each stage owes the next
Argue when RAG beats context-stuffing and fine-tuning for knowledge, and when it does not
Build a minimal end-to-end RAG loop in pure Python and trace one query through it
Name the stage a bad answer came from, given the retrieved chunks and the final output

Today's ~120 minutes

Spaced-rep warm-up: due cards from Week 16 (APIs, prompting, tools)	10 min
ELI5 + tech read; study the rag-pipeline visualizer stage by stage	25 min
Guided: build the 60-line pipeline + failure triage drill	40 min
Practice: threshold, metadata filter, and your failing query	20 min
Project: architecture one-pager	15 min
Quiz + flashcards	10 min

Builds on: Day 92 — Embeddings — meaning as geometry · Day 104 — Context windows & hallucination · Day 106 — LLM APIs — the request

The analogy

🧸 ELI5⚙ Tech

A closed-book exam tests what a student memorized months ago. An open-book exam tests something more useful: can you find the right page fast and reason from it? The student doesn't re-read the whole textbook per question — the book has an index, they look up the two relevant pages, put a finger on them, and write an answer that cites those pages.

RAG (Retrieval-Augmented Generation) turns every LLM call into an open-book exam. Instead of hoping the polymath memorized your company's leave policy during pretraining (it didn't — your docs are private and newer than its knowledge cutoff), you build the index ahead of time: split your documents into pages ("chunks"), file each one in a library with a meaning-based index (embeddings in a vector store), and at question time fetch the few most relevant pages and staple them into the prompt. The model answers *from the pages in front of it* and cites them. When the policy changes, you update the index — no retraining, and every answer can point at its source.

Why this matters on the job

RAG is the single most-built LLM system in industry, and "design a docs-QA system" is now a standard AI-engineer interview round. As an FDE it is usually the first thing a customer asks for: "can it answer questions over OUR documents?" The difference between a demo and a product is knowing where answers go wrong — retrieval missed, chunking mangled the page, or the model ignored the context — and that requires holding the whole pipeline in your head as separate, individually-debuggable stages. Your capstone, kicked off Day 119, is exactly this system taken to production.

Watch it happen

The open-book exam — one question's journey through RAG

step 1 / 5

Docspdf · wiki · tickets→Chunk~500 tokens→Embedvectors→Indexvector DB→Retrievetop-k→GenerateLLM + context

The pipeline at rest. Offline (left of the index): documents were chunked, embedded, and stored. Online: a question arrives.

🎯 Predict modespeed1×

Guided practice

guided 1

Build a whole RAG pipeline in 60 lines

25 min

Create rag_v0.py with the starter code. It is a complete pipeline: a 6-doc corpus, a bag-of-words "embedding," cosine retrieval, and a grounded prompt builder. It runs in the browser interpreter — no libraries, no API key.
Read the honesty note in the code: the word-count vector is a *lexical stand-in* for a real embedding. The architecture — embed, index, cosine top-k, assemble — is identical to production; on Day 115 you swap in real embeddings and a vector DB without touching the shape.
Run the three test queries. For each, check: did the top-2 chunks actually contain the answer?
The query "how much time off do I get" retrieves poorly — the corpus says "vacation," not "time off." Write one sentence on WHICH stage failed (retrieval: vocabulary mismatch). This exact failure is why embeddings (D115) and hybrid search (D116) exist.
Paste the printed grounded prompt into any chat LLM and confirm it answers with citations, and says "not in the context" for the out-of-scope query.

🐍 python — editable, runs in your browser

import math, re
from collections import Counter

# --- corpus: Nimbus Analytics internal docs (your capstone's world) ---
DOCS = {
    "hb-01": ("handbook", "Vacation policy: employees accrue 1.5 vacation days per month, capped at 25 days. Unused vacation days roll over, up to a maximum of 10."),
    "hb-02": ("handbook", "Parental leave is 16 weeks fully paid for all new parents, plus a 4-week flexible return-to-work period."),
    "hb-03": ("handbook", "Expense policy: individual purchases over 500 USD require manager pre-approval. Travel must be booked through the Navan portal."),
    "sec-01": ("security", "All laptops must use full-disk encryption. Report a lost device to security@nimbus.example within 4 hours."),
    "prod-1": ("product", "The Insights API rate limit is 120 requests per minute per key. Error ERR-4092 means the monthly quota is exhausted."),
    "prod-2": ("product", "Dashboards refresh every 15 minutes on the Standard plan and every 60 seconds on Enterprise."),
}

# --- "embedding": bag-of-words vector. Honest note: this is a LEXICAL
# stand-in so the pipeline runs anywhere. Real embeddings (D115) map
# synonyms near each other; this one cannot. Same architecture, weaker map.
def embed(text):
    return Counter(re.findall(r"[a-z0-9']+", text.lower()))

def cosine(a, b):
    dot = sum(a[w] * b[w] for w in a if w in b)
    na = math.sqrt(sum(v * v for v in a.values()))
    nb = math.sqrt(sum(v * v for v in b.values()))
    return dot / (na * nb) if na and nb else 0.0

# --- ingest (offline path): embed every chunk once ---
INDEX = {cid: (src, text, embed(text)) for cid, (src, text) in DOCS.items()}

# --- retrieve (online path) ---
def retrieve(query, k=2):
    qv = embed(query)
    scored = [(cosine(qv, vec), cid, src, text)
              for cid, (src, text, vec) in INDEX.items()]
    return sorted(scored, reverse=True)[:k]

# --- assemble the grounded prompt ---
def build_prompt(query, hits):
    ctx = "\n".join(f"[{cid}] ({src}) {text}" for _, cid, src, text in hits)
    return ("Answer ONLY from the context. Cite chunk ids like [hb-01]. "
            "If the context is insufficient, say 'Not in the provided context.'\n\n"
            "Context:\n" + ctx + "\n\nQuestion: " + query)

for q in ["how many vacation days roll over?",
          "what does error ERR-4092 mean?",
          "how much time off do I get",          # vocabulary mismatch!
          "what is the office wifi password?"]:  # not in corpus
    hits = retrieve(q)
    print("Q:", q)
    for score, cid, src, _ in hits:
        print(f"   {score:.3f}  [{cid}] {src}")
    print()
print(build_prompt("how many vacation days roll over?", retrieve("how many vacation days roll over?")))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Failure triage: name the guilty stage

15 min

You get four incident reports from a docs-QA bot. For each, name the guilty stage (ingest / chunking / retrieval / prompt assembly / generation) and the fix. Write your answers, then check against the key at the bottom.

Q: "What is the Enterprise refresh rate?" — Retrieved chunks: hb-01, hb-03. Answer: "Not in the provided context." (The fact IS in prod-2.)
Q: "What is the vacation cap?" — Retrieved: a chunk reading "...capped at 25 days. Unused vacation" (cut mid-sentence, no policy name). Answer is vague.
Q: "How fast must I report a lost laptop?" — Retrieved: sec-01 (contains "within 4 hours"). Answer: "Within 24 hours."
Q: "What is the refund policy?" — The refund doc lives in a PDF that the loader skipped. Answer: hallucinated a policy.

Key: 1 retrieval (right chunk exists, wrong ones returned); 2 chunking (answer split/truncated — Day 114); 3 generation (context correct, model contradicted it — needs grounding-faithfulness eval, Day 136); 4 ingest (doc never indexed; no retrieval can save you).

On your own

Extend the pipeline honestly

20 min

Extend rag_v0.py on your own: (1) add a min_score threshold to retrieve so garbage matches (score < 0.05) are dropped and the prompt says "no relevant context found" instead of stuffing noise; (2) add a source_filter argument so a query can be restricted to handbook docs only — this is metadata filtering, the baby version of per-user access control; (3) add 3 new docs and one query that your bag-of-words retriever gets WRONG but a human librarian would get right. Keep that failing query — it is your test case for Days 115–117.

Hints: threshold before slicing top-k; filter inside the comprehension; for the failing query, use synonyms the corpus never uses.

Ship before you stop

Docs-QA architecture one-pager

Write rag_architecture.md in your practice repo: a diagram-first design for a docs-QA assistant over an internal wiki (the shape of your Day 119 capstone). Draw the offline and online paths as two labeled pipelines (ASCII or Mermaid). For each of the six stages, write one line: what goes in, what comes out, and the failure mode that hides there. Close with a 5-row "symptom → suspect stage → first check" triage table built from today's guided exercise. This file becomes the skeleton of your capstone design doc.

Rubric — check what you completed (0/5)

Both paths (offline ingest, online query) drawn with all six stages in orderEach stage has input, output, and its characteristic failure modeTriage table maps at least 5 symptoms to stages with a concrete first checkStates explicitly why RAG was chosen over fine-tuning and context-stuffing for this use caseCommitted to the practice repo

Common mistakes & misconceptions

Treating RAG as one thing to debug. It is six stages; a bad answer always has ONE primary guilty stage. Debug by stage, never by re-rolling the prompt.
Embedding queries with a different model (or different preprocessing) than documents. Query and document vectors must live in the same space, produced identically.
Believing fine-tuning "teaches facts" better than RAG. FT shapes style and behavior; it cannot cite, cannot update cheaply, and cannot forget. Knowledge belongs in the index.
Skipping the "insufficient context" escape hatch. Without it, the model answers from parametric memory when retrieval fails — the worst failure, because it looks confident.
Retrieving top-k with no score threshold. When nothing relevant exists, top-k still returns k chunks of noise, and the model will dutifully answer from noise.
Ignoring metadata at ingest. Source, section, date, and access level cost nothing to store and are the only way to do filtering, citations, and permissions later.

Knowledge check

Q1. A docs-QA bot answers a question wrongly. The logs show the retrieved chunks DID contain the correct fact. Which stage is the prime suspect?

Q2. Why does RAG beat fine-tuning for keeping a company handbook answerable?

Q3. The query "how much time off do I get" fails against a corpus that says "vacation days." What failed, and what fixes it?

Go deeper — curated resources

paperRAG paper — Lewis et al. 2020 (read §1–2 for the why) ↗25 min articlePinecone Learning Center — Retrieval-Augmented Generation ↗20 min docsLlamaIndex — framework docs (high-level RAG concepts) ↗20 min

If you have a third hour

Lost in the middle — why prompt order matters — Liu et al. showed retrieval quality can be undone by placing key chunks mid-prompt. You will mitigate this on Day 117 with reranking and ordering.

Done means

rag_v0.py runs; all four test queries traced and explained
All four triage incidents attributed to the correct stage
One personally-crafted failing query saved for Days 115–117
Architecture one-pager committed
Quiz ≥ 2/3

How this connects

← Back: Day 92 gave you embeddings as a map of meaning and Day 50 gave you cosine similarity — today they became a retrieval system. Day 104 explained why stuffing the whole wiki into context fails; RAG is the engineered answer.

Forward →: Day 114 fixes the chunking stage, Day 115 swaps the toy embedding for the real thing, and Days 116–117 fix retrieval quality. On Day 119 this pipeline becomes your capstone v0, and Day 136 turns today's failure taxonomy into automated metrics.

Unlocks: D114 Chunking Strategies · D115 Embeddings & Vector Databases · D118 Advanced RAG Patterns · D119 Week 17 Checkpoint: Capstone Kickoff — Docs-QA v0