Day 141 · The tripwire

Regression Gates & CI for AI

You will be able to

Explain why prompt, model, and config changes must go through the same gate as code changes
Design an eval-in-CI gate with explicit thresholds, a canary set, and a blocking/warning split
Handle flaky evals with repeat runs, spread checks, and quarantine rules instead of deleting cases
Version prompts, model IDs, and eval data together so any regression is bisectable

Today's ~120 minutes

Spaced-rep warm-up: due cards from Week 20 (evals)	10 min
ELI5 + tech read: gates, canaries, flakiness, the version triple	20 min
Guided: build the gate script + carve the canary set	35 min
Practice: defendable thresholds with CI math	15 min
Project: wire the gate into the capstone repo + CI hook	30 min
Quiz + flashcards	10 min

Builds on: Day 135 — Graders — code, rubric & LLM-as-judge · Day 139 — Statistics for evals · Day 140 — Capstone eval harness

The analogy

🧸 ELI5⚙ Tech

A museum does not rely on a guard remembering to glance at the paintings. It strings a tripwire: anyone crossing the doorway trips it, an alarm sounds, and the door stays shut until a human clears it. Nobody debates whether tonight's intruder "seemed fine" — the wire fired or it didn't.

A regression gate is a tripwire for your AI system. Every change — a reworded system prompt, a bumped model version, a new chunk size — must walk through the doorway, and the doorway is your Day-140 eval harness running automatically. If faithfulness or retrieval recall falls below the line you drew in advance, the alarm fires and the change cannot merge. The crucial trick is the same as the museum's: you decide where the wire goes *before* anyone crosses, when you are calm, not during a heated "but the demo looked great" argument. Vibes lie; the wire doesn't.

Why this matters on the job

The most common way production AI quality dies is not a dramatic outage — it is a Tuesday prompt tweak that fixed one customer's complaint and silently broke twelve other behaviors nobody rechecked. Teams that wire evals into CI catch that in the pull request; teams that don't find out from churn dashboards weeks later. In FDE work this is a selling point: telling a customer "every prompt change runs 40 golden cases before it can merge, and here is the report" converts skeptical engineering leadership faster than any demo. Interviewers now ask "how would you stop a prompt change from regressing?" — this day is the answer.

Watch it happen

The robot release manager — a push rides the pipeline, and one gate bites

step 1 / 5

git push→Testspytest→EVAL GATEgolden set→Builddocker image→Staging→Production

✉ 42 unit tests… all green in 90 s

You push a prompt change to main. From here, no human touches the release — the pipeline decides. First stop: the classic test suite.

🎯 Predict modespeed1×

Guided practice

guided 1

Build the gate script

20 min

In your capstone repo, create evals/eval_gate.py from the starter code.
Point run_harness at your Day-140 harness (adapt the subprocess call to however yours is invoked; it must print a JSON object of metric → score).
Create evals/baseline.json by running the harness once on main and saving its output.
Run the gate: terminal: python evals/eval_gate.py — it should pass and exit 0. Check with echo $?.
Sabotage your system prompt (delete the "answer only from the provided context" line), rerun, and watch the gate fail with exit 1 and a named metric. Restore the prompt.

🐍 python — editable, runs in your browser

# evals/eval_gate.py — the tripwire. Exit 0 = merge allowed, 1 = blocked.
import json, statistics, subprocess, sys

THRESHOLDS = {          # absolute floors, chosen BEFORE the argument starts
    "faithfulness": 0.80,
    "answer_relevance": 0.75,
    "retrieval_recall": 0.85,
}
BASELINE_PATH = "evals/baseline.json"
MARGIN = 0.03           # must be below baseline by more than judge noise
RUNS = 3                # average out stochastic judges

def run_harness(case_file):
    out = subprocess.run(
        [sys.executable, "evals/run_harness.py", "--cases", case_file, "--json"],
        capture_output=True, text=True, check=True,
    )
    return json.loads(out.stdout)   # {"faithfulness": 0.83, ...}

def main():
    with open(BASELINE_PATH) as f:
        baseline = json.load(f)
    runs = [run_harness("evals/canary.jsonl") for _ in range(RUNS)]
    failures, warnings = [], []
    for metric, floor in THRESHOLDS.items():
        scores = [r[metric] for r in runs]
        mean = statistics.mean(scores)
        spread = max(scores) - min(scores)
        base = baseline.get(metric, floor)
        if spread > 0.10:
            warnings.append(f"{metric}: unstable across runs (spread {spread:.2f})")
        if mean < floor and mean < base - MARGIN:
            failures.append(f"{metric}: mean {mean:.2f} < floor {floor:.2f} (baseline {base:.2f})")
        elif mean < base - MARGIN:
            warnings.append(f"{metric}: below baseline ({mean:.2f} vs {base:.2f}) but above floor")
    for w in warnings:
        print("WARN  " + w)
    if failures:
        print("EVAL GATE FAILED")
        for fmsg in failures:
            print("  BLOCK " + fmsg)
        sys.exit(1)
    print("Eval gate passed.")

if __name__ == "__main__":
    main()

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Carve out the canary set and measure flakiness

15 min

From your ≥40-case Day-140 golden set, select 15–20 canary cases into evals/canary.jsonl. Coverage rule: at least one case per behavior (grounded answer, citation format, refusal on out-of-scope, injection resistance from D133) and every past incident you have logged.
Write a one-line rationale comment for each pick — "covers X" — in a canary_manifest.md.
Run the canary 3 times: terminal: for i in 1 2 3; do python evals/run_harness.py --cases evals/canary.jsonl --json; done.
For any per-case verdict that flips between runs, tag the case "flaky": true and note why (ambiguous rubric? judge coin-flip? genuinely borderline output?).
Decide: rewrite the rubric, or quarantine the case from blocking. Record the decision in the manifest.

On your own

Set thresholds you can defend

15 min

For each of your three gate metrics, write down: the floor, the current baseline, and — using the Day-139 bootstrap or the normal approximation — the approximate 95% CI half-width for your canary size. Then answer in writing: (1) What is the smallest real regression your gate can reliably catch? (2) Which metric would you move from blocking to warning if the team complained about false alarms, and why that one? (3) What canary size would you need to reliably detect a 5-point drop?

Hints: CI half-width for a pass rate p on n cases is roughly 1.96·sqrt(p(1-p)/n). If that is bigger than the drop you care about, the fix is more cases or repeated runs, not tighter thresholds.

Ship before you stop

Wire the tripwire into the capstone repo

Make the gate unavoidable. Commit eval_gate.py, canary.jsonl, baseline.json, and the manifest. Add a make gate target (or a gate.sh) so one command runs it locally, and document in the README that no prompt/config PR merges without a green gate. Then add the minimal CI hook: a GitHub Actions workflow file that runs the gate on every pull request (use the snippet below; Actions is taught properly on Day 152 — today you only need it to run one script). Finally, move any inline prompts into prompts/ files loaded at startup, so prompt diffs show up in PRs.

# .github/workflows/eval-gate.yml
name: eval-gate
on: [pull_request]
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install -r requirements.txt
      - run: python evals/eval_gate.py
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Rubric — check what you completed (0/6)

Gate exits 0 on main and exits 1 when the system prompt is deliberately sabotagedCanary set of 15–20 cases with a written coverage rationale per caseThresholds documented with floor, baseline, and CI-aware margin for each metricFlaky cases identified over 3 runs and quarantined or rewritten, never deleted silentlyAll prompts live in versioned files, none inline in Python stringsWorkflow file committed and visible as a check on a test PR

Common mistakes & misconceptions

Gating on a single run. One stochastic judge run can swing several points on a small set — gate on the mean of 3 runs and track the spread.
Setting thresholds after seeing the failing score. That is moving the tripwire to wherever the intruder happens to walk. Floors are set in advance and changed only by reviewed PRs.
Deleting cases that fail "unfairly". A flaky case is information — quarantine it from blocking and fix the rubric; deleting it shrinks your coverage silently.
Using a model alias like "latest" in production or evals. Provider-side upgrades change behavior under your feet and make baselines meaningless — pin exact versions.
Running the full 200-case judge suite on every push. It is slow and costly, so people start skipping it. Canary on PR, full suite nightly.
Blocking on every metric dip. If warnings never differ from blocks, engineers route around the gate. Block on floors; warn on within-noise baseline dips.

Knowledge check

Q1. Your canary pass rate drops from 0.85 to 0.82 on one run of 20 cases. The gate should…

Q2. Why must prompts, model IDs, and eval-set versions be pinned together?

Q3. A canary set exists primarily to…

Go deeper — curated resources

docspromptfoo — CI eval integration docs ↗25 min articleHamel Husain — Your AI Product Needs Evals (CI section) ↗30 min docsBraintrust — Evals guide ↗20 min docsGitHub Actions — quickstart (preview for Day 152) ↗10 min

If you have a third hour

promptfoo assertions & thresholds reference ↗ — Compare your hand-rolled gate to an off-the-shelf one: what would you keep, what would you outsource?

Done means

Gate script passes on main and blocks a sabotaged prompt (both verified)
Canary manifest committed with per-case rationale and flakiness verdicts
Thresholds written down with CI-aware margins
eval-gate.yml runs as a check on a test PR
Quiz ≥ 2/3

How this connects

← Back: This automates the Day-140 harness you built and applies Day 139's error-bar discipline to the pass/fail decision; the judge behind the scores was calibrated on Day 135.

Forward →: Day 152 folds this gate into a full test→build→deploy pipeline, Day 145 reruns the same canaries against live traffic to catch drift, and the Day-154 staging gate requires it green on every push to main.

Unlocks: D143 Logging, Feedback & the Data Flywheel · D144 Red-Teaming Lab · D145 Drift & Continuous Eval in Prod · D147 Observability Complete