Day 145 · The slow leak

Drift & Continuous Eval in Prod

You will be able to

Distinguish the drift types: model-version, data/corpus, and behavior drift after provider updates
Design online evals that sample live traffic and score it continuously
Use shadow deployments to compare a candidate against production without user risk
Set alert thresholds on quality metrics that fire on real regressions, not noise

Today's ~120 minutes

Spaced-rep warm-up: gate + flywheel cards (D141/143)	10 min
ELI5 + tech read: drift types, online evals, shadowing, alerting	20 min
Guided: continuous-eval sampler + rolling-window alarm	40 min
Practice: write the continuous-eval plan	15 min
Project: stand up continuous eval with drift demos	25 min
Quiz + flashcards	10 min

Builds on: Day 139 — Statistics for evals · Day 141 — Regression gates · Day 143 — Logging, feedback & the data flywheel

The analogy

🧸 ELI5⚙ Tech

A slow tyre leak never announces itself. The car drives fine on Monday, slightly heavy on Thursday, and by the next weekend you're on the hard shoulder wondering what happened — because nothing *happened*, it just drifted, a little each day, below the threshold where you'd notice. The fix isn't a better mechanic after the blowout; it's a pressure gauge you glance at regularly, so a slow leak shows up as a trend long before it strands you.

Your AI system leaks the same way. Nobody ships a change that breaks it. Instead the world drifts underneath it: users start asking about a new product the docs don't cover, the corpus gets re-indexed with a different chunker, or the provider silently upgrades the model behind the alias you're calling. Each nudges quality down a hair — invisible on any single answer, obvious as a two-week trend. Your Day-141 gate catches things *you* change; drift is everything you *didn't* change moving anyway. The defence is continuous evaluation: keep scoring a sample of real traffic, plot the trend, and put an alarm on the gauge so a slow leak becomes a ticket, not a 2am page.

Why this matters on the job

The scariest production AI failure is the one with no deploy attached. A customer says "it used to be great, now it's meh," your git log shows nothing, and without continuous eval you have no data to even confirm the decline, let alone explain it. Provider model updates are the classic trigger — the same prompt, the same code, different behavior overnight. Teams that sample and score live traffic see the step-change on a chart and can point to the date; teams that don't argue about vibes. For an FDE running a live customer deployment, "we caught a provider-side behavior change on the 14th and pinned the prior version within a day" is the difference between a trusted partner and a vendor on probation.

Watch it happen

The slow leak — quality drifts, the monitor catches it

step 1 / 5

weekly eval score

Your app's weekly quality score (sampled online evals, Day 145). Weeks 1–6: healthy, stable around 0.86. This baseline is what makes "worse" detectable at all.

🎯 Predict modespeed1×

Guided practice

guided 1

A continuous-eval sampler over your logs

22 min

Write obs/online_eval.py from the starter. It reads the requests.jsonl your Day-143 logging produces, samples a fraction (tail-biased: always include thumbs-downs and errors), and runs a reference-free faithfulness judge on each sampled answer against its retrieved doc IDs.
Adapt judge_faithful to call your Day-135 judge (or stub it to a keyword-overlap heuristic if you want to run offline first).
It appends a daily aggregate to obs/quality_timeseries.jsonl: date, sample size, mean faithfulness, refusal rate, feedback rate.
Run it against a few days of logged traffic (generate several days by tagging batches with different dates if needed). terminal: python obs/online_eval.py.
Confirm the time series grows one row per run — this is the pressure gauge.

🐍 python — editable, runs in your browser

# obs/online_eval.py — score a sample of live traffic, append a daily aggregate
import json, random, statistics, datetime

def load(path):
    return [json.loads(l) for l in open(path) if l.strip()]

def sample(requests, feedback_down_ids, frac=0.05):
    picked = []
    for r in requests:
        # tail-biased: always keep thumbs-down and errors; sample the rest
        if r["request_id"] in feedback_down_ids or r.get("error"):
            picked.append(r)
        elif random.random() < frac:
            picked.append(r)
    return picked

def judge_faithful(answer, doc_ids):
    # Day-135 LLM-as-judge in production; stub with your real judge call.
    # Returns 1.0 (grounded) .. 0.0 (unsupported). Placeholder heuristic:
    return 1.0 if answer and doc_ids else 0.0

def run():
    events = load("requests.jsonl")
    requests = [e for e in events if e["type"] == "ask"]
    down = {e["request_id"] for e in events
            if e["type"] == "feedback" and e["rating"] == "down"}
    picked = sample(requests, down)
    scores = [judge_faithful(r.get("answer", ""), r.get("doc_ids", [])) for r in picked]
    refusals = sum(1 for r in picked if "i can only answer" in r.get("answer", "").lower())
    agg = {
        "date": str(datetime.date.today()),
        "n_sampled": len(picked),
        "mean_faithfulness": round(statistics.mean(scores), 3) if scores else None,
        "refusal_rate": round(refusals / len(picked), 3) if picked else None,
        "feedback_down": len(down),
    }
    with open("obs/quality_timeseries.jsonl", "a") as f:
        f.write(json.dumps(agg) + "\n")
    print(agg)

if __name__ == "__main__":
    run()

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

A rolling-window alarm that does not cry wolf

18 min

Write obs/drift_alarm.py: read quality_timeseries.jsonl, keep a rolling mean of the last W days of mean_faithfulness, and raise an alert only when the rolling mean is below a floor for N consecutive days.
Parameterize W (window) and N (consecutive breaches). Start W=3, N=2, floor=0.80.
Feed it a synthetic series that dips for one day (should NOT alert) and one that declines for three days (should alert) — prove both behaviors.
Print the alert with the date the decline began, because "when did it start?" is the first question you'll ask a provider.
Note in a comment how you'd choose the floor and N from the Day-139 CI width for your sample size.

🐍 python — editable, runs in your browser

# obs/drift_alarm.py — alert on a sustained decline, not a one-day wobble
import json

W, N, FLOOR = 3, 2, 0.80          # window days, consecutive breaches, faithfulness floor

def rolling(series, w):
    for i in range(len(series)):
        window = series[max(0, i - w + 1): i + 1]
        vals = [d["mean_faithfulness"] for d in window if d["mean_faithfulness"] is not None]
        yield series[i]["date"], (sum(vals) / len(vals) if vals else None)

def run(path="obs/quality_timeseries.jsonl"):
    series = [json.loads(l) for l in open(path) if l.strip()]
    breaches = []
    for date, rmean in rolling(series, W):
        if rmean is not None and rmean < FLOOR:
            breaches.append(date)
        else:
            breaches = []
        if len(breaches) >= N:
            print(f"ALERT: faithfulness rolling mean < {FLOOR} for {N} days, "
                  f"decline began around {breaches[0]} (now {rmean:.3f})")
            return
    print("no sustained drift detected")

if __name__ == "__main__":
    run()

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Design the capstone continuous-eval plan

15 min

Write obs/continuous_eval_plan.md — a one-page plan a teammate could operate. It must specify, for your capstone: (1) which reference-free metrics you sample online and at what rate; (2) the tail-bias rule (which requests always get scored); (3) the three drift types and the concrete trigger that forces a re-eval for each (e.g. "on any model-ID change, run the full offline suite"); (4) the shadow-deployment procedure for a candidate prompt; (5) alert thresholds with the Day-139 justification for the window size. End with the escalation: what a fired alert actually pages, and the first three diagnostic steps.

Hints: a good trigger list is short and mechanical — model ID changed, corpus re-indexed, feedback rate doubled — so nobody has to remember to think.

Ship before you stop

Stand up continuous eval on the capstone

Give the capstone a working pressure gauge. Wire online_eval.py to run on a schedule (a cron entry, or a nightly GitHub Actions job — reuse the Day-141 workflow pattern) so it scores a sample of the day's traffic and appends to the quality time series. Add drift_alarm.py as a second step that exits non-zero (or prints an ALERT) on sustained decline. Simulate three scenarios and capture the gauge's response in obs/drift_demo.md: (a) steady traffic — flat line, no alarm; (b) a corpus re-index that drops retrieval quality — declining line, alarm fires with a start date; (c) a "provider swap" (change the model or degrade the judge) — step change visible in the series. For each, show the time-series rows and the alarm output.

Rubric — check what you completed (0/6)

online_eval samples tail-biased live traffic and appends a dated quality aggregateA reference-free faithfulness/groundedness signal is computed (real judge or documented stub)drift_alarm fires only on a sustained decline, not a single-day dip (both shown)The continuous-eval plan names all three drift types with a mechanical re-eval trigger eachdrift_demo.md shows the gauge responding to a corpus change and a model/judge change with start datesAlert window/threshold justified from Day-139 sample-size math

Common mistakes & misconceptions

Only running offline evals on a frozen golden set. It cannot see input drift — the users changed and your exam did not.
Alerting on instantaneous values. One bad hour is noise; alert on a rolling window over N breaches or the alarm gets muted and becomes useless.
Calling a model alias in production. Provider upgrades under the alias are invisible model-version drift — pin exact IDs and re-eval on every bump.
Sampling uniformly for online eval. Tail-bias toward errors and thumbs-downs, or you score mostly easy successes and miss the failures.
Shadowing without significance testing. A shadow that "looks better" on 20 requests proves nothing — clear the Day-139 bar before promoting.
Treating drift as a one-time audit. The whole point is continuity; a gauge you read once a quarter is a blowout waiting to happen.

Knowledge check

Q1. The same prompt and code start producing worse answers with no deploy in your git log. Most likely cause?

Q2. What is a shadow deployment for?

Q3. Why alert on a rolling window over consecutive breaches instead of a single day below the floor?

Go deeper — curated resources

articleChip Huyen — Data Distribution Shifts and Monitoring ↗30 min docsRagas — reference-free metrics (faithfulness) ↗20 min docsLangSmith — online evaluation concepts ↗15 min

If you have a third hour

Ragas continuous / production evaluation ↗ — Wire a real faithfulness judge into online_eval and compare its scores to your feedback-rate signal — do they move together?

Done means

online_eval appends a dated quality aggregate over tail-biased sampled traffic
drift_alarm fires on sustained decline and stays quiet on a one-day dip
continuous_eval_plan.md names all three drift types with mechanical triggers
drift_demo.md shows the gauge catching a corpus change and a model/judge change
Quiz ≥ 2/3

How this connects

← Back: This extends the Day-141 gate from "changes you make" to "changes the world makes," scores the Day-143 logs online, and sets alert windows with Day-139 statistics.

Forward →: Day 146 puts this time series on the dashboard, Day 147 requires the continuous-eval plan for the observability gate, and Day 157 formalizes these quality alarms into SLOs with error budgets.

Unlocks: D146 Quality & Cost Dashboards