Day 161 · Opening night

Week 23 Checkpoint: Production Cutover

You will be able to

Recall and connect the week: serving, cost, SLOs, incidents, security, and design
Cut the capstone over to a production URL with monitoring, alerts, and fallbacks live
Pass a load sanity check with concurrent users and record the latency/cost evidence
Assemble the cutover evidence pack: SLO doc, cost report, runbook, security answers

Today's ~120 minutes

Spaced-rep: full due deck for Days 155–160	15 min
Closed-book week reconstruction + diff	20 min
Cutover checklist run with evidence log	40 min
Cumulative self-quiz under exam conditions	15 min
Evidence pack assembly, tagging, issue filing	20 min
Quiz + flashcard additions from today's gaps	10 min

Builds on: Day 155 — Serving & latency budget · Day 157 — Monitoring & SLOs · Day 158 — Reliability & runbook · Day 154 — Staging pipeline

The analogy

🧸 ELI5⚙ Tech

Opening night at a theater is not the night the actors learn their lines — it is the night everything already rehearsed happens in front of a paying audience, with the stage manager watching cue sheets, understudies dressed, and a fire exit plan nobody hopes to use. The dress rehearsals this week were exactly that: you rehearsed the speed of service (Day 155), the cost of running the house (156), the alarms (157), the fire drill (158), the locks (159), and the architect's walkthrough (160).

Today the doors open: your capstone moves from "staging URL I test against" to "production URL with alarms armed and a runbook by the door." A review day also does its usual quieter work — spaced repetition. The week's ideas (percentiles, error budgets, fallback chains, the quadrilemma) share one theme: production is a system of PROMISES — to users about speed and honesty, to budgets about cost, to future-you about debuggability. Retrieval practice today is what moves those promises from notes into reflexes: you will reconstruct each artifact's logic from memory before you check it, because the exam-before-the-student-exists principle applies to your own knowledge too.

Why this matters on the job

A deployed-once project is a demo; a monitored, fallback-protected, budgeted service is a portfolio piece that interviews can probe for an hour without hitting bottom. Every artifact today maps to a real interview or customer question: "how do you know it's up?" (SLOs, canary), "what if the provider dies?" (runbook, fallback drill), "what does it cost?" (cost report), "can we trust you with data?" (questionnaire). Week 24 pivots to the FDE craft of selling and scoping this exact system — you want production truth beneath the sales story.

Guided practice

guided 1

Closed-book week reconstruction

20 min

Editor closed. On paper or a blank file, from memory:

Draw the request path with every reliability/efficiency component from this week: cache check → route decision → prefix-cached prompt → model (breaker, timeout, retries) → fallback chain → metrics/ledger writes. Annotate where TTFT starts and stops.
Write the definitions: SLI, SLO, error budget, burn-rate alert. Compute from memory: 99.5% over 30 days at 50k requests/month allows how many failures? (Check: 250.)
List the four cost levers and the quadrilemma corners.
Write the fallback chain order for a RAG app and WHY retrieval-only is a valid degraded mode.
Now open your notes and this week's files; diff. Every gap goes onto a flashcard — gaps found today are gaps absent on Day 179.

guided 2

The cutover checklist run

25 min

Execute in order, recording evidence (timestamps, screenshots, log lines) in docs/cutover_log.md:

Pre-flight: CI green on main (tests + eval gate, Day 141); staging canary green; rollback rehearsed this week (Day 158 — if not, do it now); prod secrets set via the secret store, distinct from staging.
Flip: deploy to the production target (Day 151's path). Confirm the production URL serves from outside (phone off wifi counts as "outside").
Verify: production canary green 3 consecutive runs; /metrics live; one full traced request visible end-to-end (Day 142); fallback drill — temporarily break the primary model key in prod config, watch the degraded answer + fallback counter, restore.
Load sanity: run the starter load script at 10 concurrent users for 2 minutes. Record: error count (must be 0 or honest 429s), p50/p95 latency vs your single-user baseline, cost of the run from the ledger.
Watch: 15 quiet minutes on the dashboard. Note anything surprising in the log — surprises during a calm watch are free incident prevention.

🐍 python — editable, runs in your browser

import asyncio, time, httpx

URL = "https://your-capstone-prod.example.com/ask"
QUESTIONS = ["What is the vacation policy?", "How do I submit expenses?",
             "What is the remote work rule?", "Who approves travel?"]

async def user(client, uid, results):
    for i in range(6):
        q = QUESTIONS[(uid + i) % len(QUESTIONS)]
        t0 = time.perf_counter()
        try:
            r = await client.post(URL, json={"q": q}, timeout=30)
            results.append((r.status_code, time.perf_counter() - t0))
        except Exception as e:
            results.append(("EXC:" + type(e).__name__, time.perf_counter() - t0))

async def main(concurrency=10):
    results = []
    async with httpx.AsyncClient() as client:
        await asyncio.gather(*[user(client, u, results)
                               for u in range(concurrency)])
    lats = sorted(t for s, t in results if s == 200)
    errs = [s for s, _ in results if s != 200]
    print(f"{len(results)} reqs, {len(errs)} non-200 ({errs[:5]})")
    if lats:
        print(f"p50={lats[len(lats)//2]:.2f}s p95={lats[int(len(lats)*0.95)]:.2f}s")

asyncio.run(main())

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Cumulative self-quiz, exam conditions

15 min

Ten questions, closed book, then grade with your notes. 1. Why can't you average p95s across minutes? 2. A user sees 4s of silence then fast text — which metric, which phase, two fixes? 3. Why does the semantic cache need per-ACL-group keys in the enterprise design? 4. Prefix caching: what prompt ordering exploits it? 5. Define burn-rate paging vs threshold paging and why burn wins. 6. Which errors must never be retried, and why does retry need idempotency? 7. Order the RAG fallback chain. 8. Two things that make a postmortem blameless AND useful. 9. A committed-then-deleted API key: required action and why. 10. Name the six rubric rows of a design answer.

Score honestly. Anything below 8/10: schedule the linked day's flashcards for tomorrow and re-quiz on Day 163's warm-up.

Ship before you stop

The cutover gate (capstone milestone)

Assemble the evidence pack and declare production. docs/cutover_log.md must show: production URL serving externally; canary green ×3; fallback drill executed in prod with metrics evidence; load sanity results (10 concurrent users, error count, p50/p95 vs baseline); cost of the load run reconciled to the ledger; links to the week's five artifacts (latency report, cost report, slo.md, runbook, security questionnaire). Tag the repo v0.9-production. Anything that failed its gate gets a GitHub issue with a target day (176–177 hardening window) — an honest red with a plan beats a fake green, exactly as on Day 159.

Rubric — check what you completed (0/6)

Production URL verified from outside the deployment networkCanary green three consecutive runs AND fallback drill evidence captured in prodLoad sanity: zero unhandled errors at 10 concurrent users; percentiles recorded vs baselineAll five week-23 artifacts committed and linked from the cutover logRepo tagged v0.9-production; every failed gate has an issue with a target dayClosed-book reconstruction completed with gaps converted to flashcards

Common mistakes & misconceptions

Checklist theater: ticking "runbook exists" without asking whether a stressed stranger could execute it. Every gate needs evidence, not existence.
Load-testing through the cache. Replaying four questions repeatedly measures your cache, not your service; the sanity check is honest only if you note the hit rate alongside.
Treating the load test as a scale proof. Ten concurrent users proves "no crash, graceful queueing" — claiming more invites the Day 160 interviewer to dismantle you.
Skipping the outside-the-network check. Localhost and in-VPC curls hide DNS, TLS, and firewall failures — the three most common "works for me, down for users" causes.
Reviewing by rereading notes instead of reconstructing from memory. Recognition feels like knowledge; only retrieval builds the Day 179 reflexes.
Declaring victory and closing the laptop. The first 24 hours of production are when alert thresholds prove themselves — tomorrow's warm-up includes checking what fired overnight.

Knowledge check

Q1. Under 10 concurrent users your p95 jumps from 2.1s to 6.8s while tokens/sec per request stays constant. The bottleneck is…

Q2. During the fallback drill you break the primary key and users get unlabeled cached answers with no metric change. Which TWO promises are broken?

Q3. Your error budget for the month is 250 failed requests; the load test burned 0 but the fallback drill served 40 degraded responses. Do degraded responses burn the availability budget?

Go deeper — curated resources

bookGoogle SRE Book — Ch. 4: SLOs (re-read with your own SLOs open) ↗20 min docsGrafana docs — alerting (verify your rules against best practice) ↗15 min docsDocker docs — production checklist angles ↗15 min

If you have a third hour

Write the "day 2" list — After 24 hours of production, review: which alerts fired, which were noise, what the real traffic latency histogram looks like vs the load test. Tuning thresholds against day-2 reality is the step most solo projects skip.

Done means

Reconstruction done closed-book; gaps became flashcards
Production URL live, canary ×3, fallback drill evidenced
Load sanity passed and recorded; repo tagged v0.9-production
Self-quiz ≥ 8/10 (or remediation scheduled); day quiz ≥ 2/3

How this connects

← Back: This gate stands on the whole phase: Day 148's container, Day 152's pipeline, Day 154's staging rehearsal, and this week's five promise-artifacts — speed, cost, detection, response, trust.

Forward →: Week 24 flips the table: Days 162–168 teach you to sell, scope, and demo this system as an FDE — Day 165's proposal and Day 167's demo are built on the production truth you certified today. Days 176–178 harden whatever you red-flagged.

Unlocks: D165 Proposals & Architecture Docs · D167 Demo Craft · D176 Capstone Hardening