Week 17 Checkpoint: Capstone Kickoff — Docs-QA v0
- Reconstruct the full RAG pipeline and this week's failure taxonomy from memory
- Restate the capstone brief in your own words: business context, requirements, v0 scope
- Ship capstone v0: ingest + hybrid retrieval + grounded, cited answers over a real corpus
- Score ≥ 2/3 on the cumulative Week 17 quiz and requeue weak spots
| Spaced-rep: full due deck for Week 17 (SM-2 drill) | 15 min |
| Guided 1: blank-page recall + diff + new cards | 20 min |
| Read the capstone brief; restate it in the README | 10 min |
| Guided 2: author corpus + ship v0 | 35 min |
| Practice: flowchart walk on the worst failure | 15 min |
| Cumulative quiz + requeue missed topics | 15 min |
Builds on: Day 113 — RAG architecture · Day 115 — Embeddings & vector DBs · Day 116 — Hybrid search · Day 117 — Reranking & query transforms
All week you practiced open-book exam technique on toy corpora. Today two things happen: you consolidate, and you go pro. Consolidation first, because this week's six days form ONE machine, and machines are remembered as wholes: you'll redraw the pipeline from memory, re-derive why each stage exists from the failure it prevents, and let spaced repetition do what it does — each successful recall at the edge of forgetting roughly doubles the interval before the next one, which is why today's effortful "wait, what was RRF's k again?" moment is worth ten passive re-reads.
Then the kickoff. From today until Day 180 you are building a real product for a fictional-but-realistic client. Nimbus Analytics, the 250-person company whose policy snippets you've been retrieving all week, wants an internal Docs-QA assistant. Today's deliverable is v0: the honest, minimal, working core — documents in, grounded and cited answers out. Not a demo of everything; a thin slice of the real thing, built with the exact modules you committed on Days 115–117. Every later phase of the program (evals, tracing, deployment, hardening, demo day) upgrades THIS artifact.
Shipping v0 early is the FDE method: a working thin slice beats a perfect plan, because every later decision (evals, observability, deploys) needs a real system to attach to. This capstone is also your portfolio centerpiece — by Day 180 it will be the repo you walk interviewers through, with a commit history that tells a 60-day production story. And the review half matters just as much: Week 18 (agents) builds directly on retrieval-as-a-tool, so Week 17's machinery must be recallable without notes before you hand it to an agent.
The open-book exam — one question's journey through RAG
step 1 / 5The pipeline at rest. Offline (left of the index): documents were chunked, embedded, and stored. Online: a question arrives.
Guided practice
Blank-page recall: rebuild the week from memory
20 minClose every note and editor tab first.
- On paper or a blank file: draw the six-stage RAG pipeline with offline/online paths, and annotate each stage with its characteristic failure (D113).
- Write the BM25 scoring idea (IDF × saturated TF × length norm) and the RRF formula with its k, from memory (D116).
- Write three sentences: why a cross-encoder outranks a bi-encoder, and why it can't replace stage one (D117).
- List the three chunking families and your decision-card defaults (D114), and the embedding-model lock-in rule (D115).
- NOW open your notes and diff. Every gap or error goes onto a flashcard — gaps found today are the highest-value cards you'll write this month.
- Re-answer, from memory, the two quiz questions you found hardest this week (check the dashboard's missed-question list).
Author the Nimbus corpus, then ship v0
35 min- Create the capstone repo:
docs-qa/withcorpus/,src/,tests/, and a README containing the brief above in your own words (5 sentences — this is your Day 165 proposal seed). - Author the corpus: 12–20 markdown files under
corpus/. Include on purpose: one table (per-diem rates), one pair of near-duplicate docs (old + new expense policy, the old one marked superseded), API docs with error codes, and at least two facts that require different sections to answer. Plant 3 "landmine" facts you'll use to test refusals and freshness later. - Wire the pipeline: copy in your
build_index.py(D115),retrieval.pywith reranker (D116–117). Point ingest atcorpus/, build both indexes. - Write
answer.py:answer(question) -> {text, citations, refused}. Assemble the grounded prompt (D113's contract: cite chunk ids, refuse on insufficient context, best chunk first per D117's deep-dive). If you have an API key, call your Day 107 client wrapper; otherwise print the assembled prompt and paste into a chat model — the architecture is what's graded today. - Write
tests/smoke.py: 10 questions with expected source sections — 6 straightforward, 2 requiring hybrid (error codes), 1 refusal case (not in corpus), 1 against the superseded-policy pair. Run it; record the score in the README.
On your own
Walk your own flowchart
15 minAt least one of your 10 smoke tests failed (if not, add harder ones until one does — a suite that can't fail teaches nothing). Take the worst failure and walk your Day 118 rag_debugging.md flowchart on it, top-down, writing one line per check: ingested? intact in one chunk? in stage-one top-20? in reranked top-3? in the prompt? used faithfully? Fix the failure at the guilty stage, re-run the suite, and append a 3-line postmortem (symptom → stage → fix) to the README. This is the debugging ritual you'll repeat for 60 days.
Capstone v0: grounded docs-QA over the Nimbus corpus
Deliver the v0 slice end-to-end in the new docs-qa repo: authored corpus (12–20 docs with the planted structures), ingest with metadata + stable IDs, hybrid retrieval + reranking, and answer() returning grounded text with structured citations or an honest refusal. The smoke suite (10 questions) runs with one command and the README records the score, the brief-in-your-own-words, and one postmortem from the flowchart walk. Tag the commit v0. This repo now receives every upgrade through Day 180 — treat the commit history as part of the portfolio.
Common mistakes & misconceptions
- Building v0 features that are scoped out (UI, auth, dashboards). The discipline of "out means out, with a scheduled day" is the FDE scoping muscle — v0 is a thin slice, not a small everything.
- Grabbing a random public corpus instead of authoring one. You need to KNOW where every answer lives to diagnose failures; authored landmines make Weeks 20–21 evals meaningful.
- Reviewing by re-reading the week's lessons. Recognition feels like knowledge; only retrieval from a blank page reveals and repairs the gaps. Diff after recalling, never before.
- Letting the smoke suite pass 10/10 on day one. If it can't fail, it can't teach — add the refusal case and the superseded-policy case until something breaks, then flowchart it.
- Skipping the honest-refusal path because "the corpus covers everything." The named client nightmare is wrong-but-confident policy answers; refusal is a feature with a test, not an apology.
- Deferring stable IDs and metadata "until it matters." It matters on Day 145 (freshness) and Day 133 (injection-via-docs) — and retrofitting provenance is a full re-ingest.
Q1. Your v0 answers a question about the OLD expense policy (superseded doc). Which two mechanisms, both from this week, address this?
Q2. A smoke test on "ERR-4092" fails under dense-only retrieval but passes with your hybrid module. This is because…
Q3. Why does the v0 prompt place the best reranked chunk FIRST in the context?
Go deeper — curated resources
- Write the v0 announcement — Optional FDE rep: draft the 5-sentence Slack message announcing the pilot to Nimbus employees — what it does, what it won't do yet, where to report bad answers. You'll reuse it on Day 167 (demo craft).
- Blank-page recall diffed; every gap converted to a flashcard
- docs-qa repo created, corpus authored, v0 tagged
- Smoke suite ≥ 8/10 with one postmortem written
- Brief restated in your own words in the README
- Cumulative quiz ≥ 2/3 (retake after targeted review if lower)
← Back: Everything this week converged here: D113's architecture, D114's decision card, D115's build_index.py, D116–117's retrieval.py, D118's flowchart — v0 is those five artifacts composed.
Forward →: Day 120 hands retrieval to an agent as a tool. Day 140 builds v0's golden-set eval harness, Day 142 adds tracing, Day 148 containerizes it, Day 154 gives it CI, and Day 180 you demo it. The repo you tagged today is the one you'll present.
Unlocks: D132 Guardrails, Injection & AI Security · D133 Week 19 Checkpoint: Red-Team Your RAG · D134 Eval Mindset & Golden Sets