AI System Design
- Run the AI-system-design method: requirements → constraints → architecture → deep dives → trade-offs
- Navigate the quality/latency/cost/privacy quadrilemma explicitly in every design decision
- Reproduce three complete designs: a support copilot, enterprise doc-QA, and an agent with approvals
- Defend component choices (RAG vs fine-tuning, model tiers, sync vs async) against follow-up probes
- Self-grade a design answer against a rubric interviewers actually use
| Spaced-rep warm-up: due cards incl. Day 48 method & Day 123 workflows | 10 min |
| ELI5 + tech read; write the method skeleton + quadrilemma from memory | 15 min |
| Guided: designs 1 and 2, cover-and-reproduce protocol | 40 min |
| Practice: design 3 attempted cold, then graded against the model | 25 min |
| Project: portfolio entry + rubric self-assessment | 20 min |
| Quiz + flashcards | 10 min |
Builds on: Day 48 — System design method · Day 123 — Multi-agent & workflow patterns · Day 136 — RAG evaluation · Day 158 — Reliability & fallback chains
An architect's licensing exam doesn't ask "do you know what a beam is?" It hands over a messy brief — "a clinic, on this sloped site, this budget, must open by winter" — and watches HOW you work: do you ask what the clinic actually needs before drawing? Do you say out loud "I'm choosing wood over steel because of budget, accepting a height limit"? Do you know where your own design is weakest? A candidate who draws a gorgeous building for the wrong brief fails; one who draws a modest building with clearly reasoned trade-offs passes.
The AI system design interview — now a standard round for AI engineer and FDE roles — is exactly this exam, and so is every real customer engagement. The brief is always underspecified on purpose. The winning behavior is a loop you can rehearse: pin down requirements and scale, name the constraints that actually bind (in AI systems it is almost always some corner of quality-latency-cost-privacy), draw a boring-but-right architecture, then go deep where the risk lives — retrieval quality, eval strategy, failure modes. Today you study three model answers the way a chess student studies annotated grandmaster games: not to memorize them, but to absorb the move patterns.
This round decides senior-vs-junior offers. Coding rounds test whether you can build a component; the design round tests whether you can be left alone with an ambiguous, expensive problem — which is the actual job, and doubly so for FDEs who do this live in front of customers (Day 168's simulation is this skill with a customer in the room). The 2026 twist: interviewers now expect evals, guardrails, and cost engineering IN the design, not bolted on when prompted. Your last 60 days are precisely those muscles; today assembles them into an interview-shaped performance.
The architect's exam — a support copilot, assembled piece by piece
step 1 / 5Start with the spine every AI system shares: client → API → model. It works — and it's naive.
Guided practice
Worked design 1 — customer-support copilot (agent-assist)
20 minRead this model answer in full, then close it and reproduce the architecture and both deep dives from memory; diff against the original.
Brief: "Design an AI copilot for our 300-agent customer-support team." (Deliberately vague.)
Requirements (asked, not assumed): Who uses it — agents, not end customers? (Yes: human-in-the-loop, which lowers risk tolerance needed.) What does it do — suggest replies, summarize tickets, surface KB articles? (All three; suggested replies are the core value.) Volume — 300 agents × ~40 tickets/day = 12k tickets/day, ~3 model interactions each = ~36k requests/day, peak ~10/sec. Latency — suggestions must land while the agent reads the ticket: TTFT under ~1.5s, streaming. Privacy — tickets contain customer PII; retention and redaction matter. Quality bar — a bad suggestion wastes agent time but a human reviews everything: medium risk, favor speed and cost.
Quadrilemma call: latency and cost weighted highest; quality medium (human review); privacy handled by redaction + provider terms. This licenses a mid-tier model.
Architecture (spoken as a request walkthrough): Ticket event → ingestion service normalizes and redacts PII → context builder assembles: ticket thread summary (cached per ticket, updated incrementally), customer metadata from CRM (read-only API), and top-k KB chunks via hybrid retrieval (BM25 + dense, reranked — Days 116–117) → prompt with stable prefix (system + few-shots) for prefix caching (Day 156) → mid-tier model, streamed into the agent desktop as three ranked suggested replies with KB citations → agent edits/accepts/rejects → the accept/edit/reject signal is logged as feedback (Day 143) — this flywheel is the design's best feature: every agent action is a free label.
Deep dive 1 — KB retrieval: KB articles chunked structure-aware (Day 114), nightly full re-index plus event-driven upserts on article edits; retrieval eval = precision@5 on a golden set mined from historical ticket→article links agents already click. Deep dive 2 — evals & rollout: offline golden set of 200 historical tickets with accepted replies; LLM-judge for helpfulness calibrated against agent labels (Day 135); online metric = suggestion acceptance rate; ship behind a 10%-of-agents canary; regression gate in CI on the golden set (Day 141).
Trade-offs stated: chose suggestion-only over auto-send (quality risk unacceptable without review); mid-tier model over frontier (latency+cost; acceptance rate will tell us if we underbought); RAG over fine-tuning for KB knowledge (KB changes weekly; fine-tuning bakes stale knowledge — Day 127's decision tree); accepted a cold-start weakness: acceptance rate is meaningless until agents trust the tool, so weeks 1–2 measure edit distance instead.
Worked design 2 — enterprise doc-QA at scale (your capstone, times 100)
20 minSame protocol: read, cover, reproduce the envelope math and the freshness deep dive, diff.
Brief: "A 40,000-employee company wants employees to ask questions over 2M internal documents — policies, wikis, contracts — with per-document access control."
Requirements: The two words that reshape everything: access control. An employee must never see an answer derived from a document they cannot open. Also: 2M docs ≈ (say) 40M chunks; usage assume 20% of employees ask 2 questions/day = 16k questions/day, peak 30/sec; freshness — HR policies must reflect edits within hours; audit — legal wants to know who asked what (retained, access-controlled, Day 159).
Quadrilemma call: privacy is the binding corner (ACL correctness is non-negotiable), quality second (wrong policy answers create liability), then latency, then cost.
Envelope math (out loud): 16k questions/day × ~4k tokens in (context-heavy RAG) + 300 out ≈ 70M tokens/day ≈ low-hundreds of dollars/day at mid-tier prices — real money: caching and routing (Day 156) are design requirements, not nice-to-haves.
Architecture: Ingestion is a first-class subsystem, not a script: connectors (SharePoint, Confluence, drive shares — Day 169 foreshadow) → parsing/chunking workers on a queue (Day 47) → embeddings → vector DB with ACL metadata on every chunk (source doc ID, allowed groups) plus BM25 index. Query path: authenticate → resolve user's groups → hybrid retrieval with ACL pre-filter in the query — filter at retrieval time, never post-filter after the model has seen forbidden text → rerank → generate with mandatory citations → citation validator checks every cited doc against the user's ACLs again (defense in depth) → respond, streamed. Semantic cache keyed per ACL-group, never global — a cached answer derived from restricted docs must not leak to a broader audience.
Deep dive 1 — freshness: event-driven upserts from source-system webhooks where available, else delta-crawl every N hours; tombstone deletions immediately (a revoked doc must leave the index NOW — that is a security event, not an eventual-consistency shrug); staleness SLI: p95 time from doc-edit to index-visibility. Deep dive 2 — the ACL failure mode: the nightmare is retrieval-time filter drift (group membership changed, index metadata stale), hence the second ACL check at citation time and a red-team eval (Day 144): a golden set of "user X asks about doc they cannot read" cases that must ALL refuse — gated in CI.
Trade-offs: pre-filter retrieval can hurt recall for users with narrow access (accepted; correctness beats completeness — and stated so the customer decides); per-group caching slashes hit rate (accepted for the same reason); chose managed vector DB over self-hosted at 40M chunks unless data-residency forces otherwise (ops cost dominates); fine-tuning rejected again — 2M living documents is the textbook RAG case (Day 113).
On your own
Worked design 3 — attempt first: an agent with approvals
25 minBrief: "Design an AI agent that processes vendor invoices: read the invoice, match it to a purchase order, flag discrepancies, and schedule payment — with human approval where it matters." Attempt the full answer solo (requirements, envelope, architecture, two deep dives, trade-offs) in 20 minutes, writing as you would speak. THEN read the model answer below and grade yourself with the project rubric.
---
Model answer. Requirements: volume — say 3k invoices/month, latency irrelevant (batch, minutes are fine — this changes everything: no streaming, cheap async workers); accuracy critical — money moves; audit trail legally required; the approval boundary is THE design question. Quadrilemma: quality and auditability dominate; latency nearly free; cost minor at this volume.
Architecture: event-driven pipeline (Day 123's workflow-not-agent insight — the steps are known, so use a workflow with LLM steps, not a free-roaming agent): invoice arrives → document extraction to a strict schema (Day 110 structured outputs + Day 130 document AI), validators on totals/dates/currency — retry loop on validation failure → deterministic PO matching first (exact PO number), LLM assist only for fuzzy cases, with match confidence → discrepancy rules engine (price/quantity deltas beyond tolerance) → approval gates by risk tier: auto-approve exact matches under $1k (still logged); human approval queue for fuzzy matches, discrepancies, or >$10k; ALL payment scheduling requires an idempotency key (Day 46 — retried payments must never double-pay). Every step writes to an append-only audit log: inputs, model version, prompt version, confidence, human decision.
Deep dive 1 — extraction quality: golden set of 100 real invoices (multi-format, scanned and native PDF), field-level accuracy metrics, hard gate in CI; per-field confidence drives routing to human review, and the human corrections feed the golden set (flywheel again). Deep dive 2 — the approval UX: reviewers see the extracted fields NEXT TO the source image with differences highlighted — reviewer throughput is the real bottleneck at scale, and a bad review UI silently becomes rubber-stamping; measure reviewer disagreement rate as a canary for extraction drift (Day 145).
Trade-offs: workflow over agent (predictability and auditability beat flexibility; an open-ended agent moving money is an unforced error); auto-approval threshold is a business decision surfaced to finance, not hidden in code; chose two-model cascade (cheap extractor, strong model only on low-confidence fields) for cost; accepted slower processing for fuzzy cases because reviewer quality beats speed here.
Self-grade honestly: where did your attempt diverge — and was the divergence a defensible alternative (fine) or a missed requirement (drill it)?
Your design portfolio entry + the rubric
Create docs/design_answers.md: (1) the rubric below, kept verbatim for reuse on Days 168 and 179; (2) your reproduced versions of designs 1–2 (from the cover-and-reproduce drills) tightened to one page each; (3) your graded attempt at design 3 with a self-assessment paragraph naming your two weakest rubric rows and one drill for each. Rubric (score each 0–3): requirements — asked clarifying questions, quantified scale; constraints — named the binding quadrilemma corner with justification; architecture — complete request walkthrough incl. ingest/write path; evals — measurement plan with golden set and gate; reliability — failure modes and fallbacks unprompted; trade-offs — explicit sacrifices with alternatives named. 14+/18 is a passing interview performance.
Common mistakes & misconceptions
- Drawing architecture before asking a single question. The brief is vague ON PURPOSE; the first five minutes of requirements questions ARE the assessment.
- Maximalism: agents + fine-tuning + multi-model routing for a problem RAG and a prompt solve. Boring-but-right beats impressive-but-unjustified; reach for Day 120's when-not-to-build-an-agent checklist.
- Designing only the read path. Where does the corpus come from, how does it update, what happens on deletes? The ingest/freshness story is where enterprise designs live or die.
- Claiming quality with no measurement plan. "It will answer accurately" is a vibe; a golden set, a calibrated judge, and a CI gate is a design.
- Post-filtering ACLs after retrieval. If forbidden text reached the prompt, you already leaked — filter at retrieval time and re-check at citation time.
- Hiding trade-offs. Interviewers and customers both trust the person who says "I sacrificed X for Y; if you weight X more, here is the alternative" — naming costs is seniority.
Q1. In the enterprise doc-QA design, why must ACL filtering happen AT retrieval time rather than on the final answer?
Q2. The invoice system was designed as a fixed workflow with LLM steps rather than a free-roaming agent because…
Q3. An interviewer asks "how do you know your support copilot is good?" The strongest answer is…
Go deeper — curated resources
- Drill: 10-minute lightning designs — Prompt yourself with one-liners (meeting summarizer for a hospital; code-review bot; multilingual FAQ) and produce ONLY requirements + quadrilemma call + one-paragraph architecture. Speed-reps build the reflex the full designs refine.
- Designs 1–2 reproduced from memory and diffed against the models
- Design 3 attempted before reading its answer, scored on all six rubric rows
- design_answers.md committed with self-assessment and drills
- Quiz ≥ 2/3
← Back: This is Day 48's method with sixty days of new vocabulary plugged in: Day 113's RAG pipeline, Day 123's workflow-vs-agent judgment, Day 134's eval mindset, Day 156's cost levers, Day 158's fallbacks — spoken as one coherent answer.
Forward →: Day 168 runs this skill with a customer in the room instead of an interviewer, Day 165 turns designs into sellable documents, and Day 179's interview gym re-runs a full mock against today's rubric.
Unlocks: D162 The FDE Role · D165 Proposals & Architecture Docs · D171 Stakeholders & Trade-off Navigation · D179 Interview Gym