Capstone Quality Gates
- Re-run the full eval harness and produce a current, honest eval report with confidence intervals
- Re-execute the Day 133 and Day 144 attack suites against the hardened system and close or ticket every finding
- Verify tracing, load behavior, and the security checklist as pass/fail gates, not impressions
- Produce the final cost analysis from measured production numbers
- Publish a gate scorecard where every claim links to evidence
| Punch-list review + commit thresholds table | 10 min |
| Read: the six gates and the rules | 10 min |
| Gates 1–2: evals + attack suites | 25 min |
| Gates 3–4: tracing + load re-verify | 25 min |
| Gates 5–6: security walk + measured cost | 25 min |
| Scorecard finalize, fixes/tickets, quiz | 20 min |
Builds on: Day 140 — Capstone eval harness · Day 144 — Red-teaming lab · Day 176 — Capstone hardening
After punch-list week, the building inspector arrives. She does not ask the builder "is the wiring good?" — she flips every breaker herself. She doesn't admire the fire escape; she puts weight on it. Her clipboard has a fixed list, each item is pass or fail, and "we fixed that last week" counts for nothing without a re-test in front of her eyes. Builders who are annoyed by inspectors build worse buildings; builders who internalize the inspector build things that don't burn down.
Today you are your own inspector, and the rule of the day is: no claim without a fresh test. Yesterday you hardened the doors — today you try to kick them in again, using the same attack suites that broke the system back on Days 133 and 144. The eval harness runs again, from scratch, and whatever score comes out IS the score that goes in the report — error bars and all. The load check runs again. The cost numbers come from the meter, not the estimate. What you end the day with is rarer than a good system: a system whose goodness is DOCUMENTED, test by test, in a scorecard a stranger could audit.
"How do you know your system works?" is the question that separates senior candidates from juniors in every AI engineering interview — and the only convincing answer is machinery: golden sets, attack suites, gates, reports with error bars. Customers run this exact inspection on vendors (Day 159's security checklist is their clipboard); doing it to yourself first is how FDEs walk into security reviews relaxed. And tomorrow you ship v1.0 — the difference between "shipped" and "shipped, with evidence" is today.
Guided practice
Gate 1–2: evals and the attack suites
25 min- Write the thresholds table FIRST in
docs/quality-gates.md— all six gates, numbers committed before any run. - Run the full eval harness on the current build. Generate the report: per-metric scores WITH bootstrap CIs (reuse your Day 139/140 code), plus the three worst-scoring cases examined by hand — worst cases are tomorrow's talking points, know them cold.
- Compare against your Day 140 baseline: any metric down beyond noise (CI overlap test) is a regression introduced by hardening week — find it (suspect: the new validation caps truncating context, or the refusal path triggering on answerable questions).
- Re-run the Day 133 + Day 144 attack suites end-to-end. Log each attack: blocked / refused / SUCCEEDED. Any success is a P0 for today.
- Write and run three NEW attacks the old suites lack — candidates: injection via a document uploaded through the hardened ingest path, an exfiltration attempt against the new failure messages (do they leak internals under weird inputs?), a rate-limit-evasion probe across two user accounts.
Gate 3–4: tracing and load, re-verified
25 min- Tracing: exercise five distinct paths (normal Q&A, refusal, cached answer, ingest, induced provider failure). For each, pull the trace and check: complete span tree, latency per stage, token counts present, PII/redaction rules intact on stored payloads. Record the five trace IDs in the scorecard as evidence.
- Load: re-run the Day 161 concurrency check against the hardened build (same tool, same N, same corpus — a changed benchmark is no benchmark). Record p50/p95, error rate, and cost of the run.
- Push PAST the rate limit deliberately: verify the system sheds with 429 + Retry-After and stays healthy for other users — the graceful-degradation gate the hardening added.
- While the load run is hot, compute Gate 6 inputs: mean input/output tokens per request from the traces — the measured numbers the cost gate needs.
On your own
Gate 5–6: security walk and measured cost
25 minNo steps — the checklists are yours. Walk Day 159's security checklist item by item against the current system, marking each with evidence or a ticket (git history secret scan, TLS config, dependency audit via pip-audit or equivalent, log-redaction spot check on real logs, yesterday's perimeter suite re-run). Then finalize the cost gate: measured cost/query from the load run's token counts × current prices, projected monthly at the one-pager's volume assumptions; if it misses the Day 173 one-pager by > 20%, correct the one-pager and note why the estimate was off — the miss analysis is itself Demo Day material.
Constraints: every checklist item gets evidence or a ticket, no bare checkmarks; timebox 25 minutes — items that need real work become tickets, not heroics.
Hint: the classic finding at this stage is a dependency with a known CVE and a redaction rule that misses the NEW failure-message logs added yesterday.
The gate scorecard
Complete docs/quality-gates.md: six gates, each with pre-committed threshold, measured result, PASS/FAIL, and an evidence link (eval report file, attack log, trace IDs, load output, checklist, cost calc). Attach the regenerated eval report (with CIs) and the attack log as committed artifacts. Every FAIL is either fixed-and-re-run today or ticketed with severity and a one-line Demo Day disclosure. Finish by updating the punch list: D177 items marked DONE with hashes. Tomorrow ships whatever this scorecard says is true — so make it true or make it honest.
Common mistakes & misconceptions
- Setting thresholds after seeing results. That is decoration, not gating — the commit history should prove the thresholds came first, exactly like Day 134's exam-before-the-student principle.
- Re-running only the attacks that failed last time. Regressions hide in the ones that used to pass; suites run whole or they are anecdotes. And a suite that never gains new attacks decays with the threat landscape.
- Reporting point estimates. A groundedness of 0.91 vs a gate of 0.90 with ±0.04 of bootstrap noise is not a pass — it is "indistinguishable from the gate"; say so and decide consciously (Day 139).
- Benchmarking load against a different corpus, N, or tool than Day 161. Changing the benchmark mid-program destroys the comparison; the whole value of a re-check is held-constant conditions.
- Estimated cost in the "measured" gate. The traces contain real token counts; using the old estimates defeats the gate's purpose — and the estimate-vs-measured miss is often the most interesting finding of the day.
- Hiding the one red gate. An all-green scorecard reads as unaudited; a scorecard with one honest ticket and a disclosure line reads as engineering. Integrity is the demo.
Q1. Why must gate thresholds be committed before running the tests?
Q2. Your eval score is 0.91 against a 0.90 gate, with a bootstrap CI of ±0.04. The honest reading is:
Q3. The re-run load test must use the same tool, N, and corpus as Day 161 because:
Go deeper — curated resources
- Braintrust evals guide — CI-integrated eval reports ↗ — Compare their report structure with yours; steal the good ideas for the shipped report.
- Thresholds table committed before first run (verifiable in git history)
- All six gates executed with evidence links; eval report has CIs
- Three new attacks logged; zero unresolved attack successes without tickets
- Cost corrected or confirmed against measured tokens; quiz ≥ 2/3
← Back: Day 140 built the harness, Days 133/144 built the attack suites, Day 139 supplied the error bars, Day 161 set the load baseline, Day 159 wrote the security clipboard — today ran them all against the hardened system as one inspection.
Forward →: Tomorrow (Day 178) the scorecard ships inside v1.0 as the evidence section of your docs. On Day 180, the numbers on this scorecard are the numbers in your presentation — and the open tickets are your prepared answers for hard questions.
Unlocks: D178 Capstone Ship & Document