Week 19 Checkpoint: Red-Team Your RAG
- Recall Week 19 concepts (FT decision, LoRA, quantization, multimodal, injection) from memory
- Run a structured 15-attack suite against your capstone v0 and log outcomes reproducibly
- Triage findings by severity and fix the top holes with layered defenses
- Write a findings memo a security reviewer and a non-technical stakeholder can both read
- Turn every attack into a reusable test case for the future eval and CI gates
| Spaced-rep: due Week 19 deck (all six lessons) | 10 min |
| Closed-book recall of Week 19 + diff against notes | 20 min |
| Build and run the 15-attack suite | 35 min |
| Practice: fix top holes and prove closure | 20 min |
| Project: write the findings memo | 20 min |
| Cumulative quiz + write new flashcards from gaps | 10 min |
Builds on: Day 119 — Capstone v0 — docs-QA to attack · Day 127 — Fine-tuning decision · Day 132 — Injection, trifecta & defense-in-depth
Before a museum opens the new wing, it hires people to try to steal the paintings — on purpose, on a Tuesday, with the curator watching. Better a friendly thief finds the unlocked window on a quiet morning than a real one finds it on opening night. Today you are the friendly thief, and the museum is your own Docs-QA service.
The value isn't the drama of breaking in; it's the list you carry out. Every window you found unlocked becomes a work order, and — the part beginners skip — a permanent tripwire, so if someone re-opens that window next month an alarm goes off automatically. A red-team you run once and forget is theater. A red-team whose every finding becomes a test case that runs forever is engineering. This week also asks you to prove, from memory, that you can still draw the fine-tune decision tree and explain the lethal trifecta — recall, not recognition.
"Have you tried to break it yourself?" is a question every serious buyer and every senior interviewer asks, and "yes, here is my findings memo and the regression tests that keep the holes closed" is one of the strongest answers a junior engineer can give. The memo itself is FDE muscle: translating "your indirect-injection surface allows exfiltration via retrieved documents" into a risk a stakeholder can weigh and fund. Doing this before Day 140's eval harness means your security cases are already in the test set when you automate.
Guided practice
Closed-book recall of Week 19
15 minClose all notes. On a blank page, reproduce from memory:
- The prompting vs RAG vs fine-tuning decision tree, with the deciding question at each branch.
- The LoRA equation and the trainable-parameter count for a d×k layer at rank r.
- The memory footprint of a 7B model at fp16 / int8 / int4, and the one-line rule for when 4-bit is safe.
- When you choose OCR-first vs VLM-native for documents.
- The lethal trifecta's three legs and one way to break it.
- Two prompt-injection payload styles and one defense per layer (input/tool/output/architecture).
Then open Days 127–132 and diff. Grade each item; anything you missed becomes a flashcard you write today. Recall you cannot reproduce is recognition you overestimated — this diff is the whole point of a review day.
Build and run the 15-attack suite
30 min- Create
redteam/suite.py. Encode the 15-attack taxonomy as a list of cases: each a dict with id, category, severity, payload (a question and/or a poisoned document to inject into the corpus), and expected_safe (a predicate describing safe behavior). - Wire it to your capstone's answer function (or the Day 132 toy RAG if your v0 isn't ready). For document-borne attacks, insert the poisoned doc into the corpus before the query and remove it after.
- Run the whole suite; for each case, auto-check the cheap signals (did the banned string appear? did it answer a no-context question? did an outbound URL show up?) and mark SAFE/VULNERABLE/PARTIAL. Print a category summary.
- Save results to
redteam/results.jsonwith timestamps — this is a reproducible artifact, not a one-off. - Manually review the PARTIALs: automated checks miss nuance (Day 135 will formalize this). Adjust labels with a note.
On your own
Fix the top holes and prove closure
20 minTake your two highest-severity VULNERABLE findings. For each: apply a Day-132 defense at the right layer, then re-run the EXACT suite and show the case flips to SAFE while no previously-SAFE case regresses.
Deliver a before/after results diff (VULNERABLE → SAFE for the fixed cases, everything else unchanged) and, for each fix, one sentence on which layer you used and why that layer (not another) was the right place. If a fix closes the hole but breaks a legitimate query (over-blocking), record that too — security that destroys utility is its own failure.
Hints: exfiltration is usually best killed at output/architecture (strip outbound URLs, deny the channel), not by pleading with the model in the prompt. Re-run the whole suite, not just the fixed case — regressions hide in the cases you didn't look at.
Capstone red-team findings memo
Write docs/redteam-findings-w19.md in the capstone repo — the deliverable a customer's security team and your own PM will both read. Sections: (1) scope and method (what "compromise" meant, how many attacks, reproducibility — reference redteam/suite.py); (2) results table (id, category, severity, verdict) with a one-line executive summary a non-engineer understands ("no critical vulnerabilities remain open; two were found and fixed"); (3) the fixes applied, by defense layer, with before/after evidence; (4) accepted risks and open items, ticketed; (5) the plan to keep these closed — every attack becomes a Day 140 eval case and a Day 141 CI gate. Commit the memo, the suite, and results.json.
Common mistakes & misconceptions
- Improvising attacks instead of enumerating a taxonomy. Ad-hoc pokes miss whole categories; a structured suite is reproducible and complete.
- Counting "how many passed" instead of "are any CRITICAL open." One unfixed exfiltration hole outweighs twelve blocked system-prompt pokes.
- Fixing at the wrong layer — pleading with the prompt to stop exfiltration. Kill the channel (output/architecture); prompts are bypassable.
- Fixing without re-running the whole suite. Defenses cause regressions and over-blocking; only a full re-run proves closure without collateral damage.
- Running the red-team once. Attacks evolve and code changes; unrepeated security is theater. Institutionalize into evals (Day 140) and CI (Day 141).
- Treating a review day as re-reading. Recognition feels like mastery and isn't; the closed-book diff is where the real gaps surface.
Q1. Your red-team finds 12 SAFE and 3 VULNERABLE. The 3 are: reveal-system-prompt (low), off-policy answer (medium), exfiltrate-via-URL (critical). What do you fix first?
Q2. Why must every red-team finding become a permanent eval/CI case?
Q3. From memory: a 7B model in int4 needs roughly how much weight memory, and LoRA at rank 8 on a 4096×4096 layer trains how many params?
Go deeper — curated resources
- Attack libraries and automated red-teaming — Tools like promptfoo's red-team and garak generate large adversarial suites automatically. Useful for breadth, but hand-built cases tied to YOUR trust boundaries catch the vulnerabilities that matter — Day 144 combines both.
- Closed-book recall completed and diffed; gap flashcards written
- Reproducible 15-attack suite runs and writes results.json
- Top-two findings fixed with before/after evidence, no regressions
- Findings memo committed with executive summary + ticketed open risks
- Cumulative quiz ≥ 2/3
← Back: This puts Day 132's attacks and defenses to work on Day 119's real capstone, and drills the whole week (127–132) from memory. The severity discipline echoes Day 44's security thinking.
Forward →: Every attack case seeds the Day 140 eval harness and the Day 141 CI gate. Day 144 is a deeper formal red-team, and Day 177 re-runs this exact suite as a release gate before Demo Day.
Unlocks: D144 Red-Teaming Lab