Serving & Inference Optimization
- Decompose LLM latency into TTFT, tokens/sec, and total time, and measure each
- Explain why continuous batching beats static batching for LLM serving
- Describe what the KV cache stores and why it makes generation memory-bound
- Compare self-hosted (vLLM-class) serving against hosted APIs on cost, control, and ops burden
- Produce a written latency budget for your capstone with p50/p95 numbers
| Spaced-rep warm-up: due cards from Week 22 (Docker, CI/CD, IaC) | 10 min |
| ELI5 + tech read; sketch the prefill/decode split from memory | 20 min |
| Guided: latency probe + batching simulation | 40 min |
| Practice: latency budget worksheet | 20 min |
| Project: capstone latency report with one shipped improvement | 20 min |
| Quiz + flashcards | 10 min |
Builds on: Day 95 — The Transformer — causal masking & per-token generation · Day 107 — LLM APIs II — streaming, retries & cost · Day 151 — Deploy Lab — container to cloud URL
Watch a drive-through at lunch rush. The metric customers feel first is not "how long until my whole order is ready" — it is "how long until someone talks to me at the speaker." A drive-through that greets you in five seconds and hands out food steadily feels fast even when the full order takes three minutes. One that leaves you in silence for ninety seconds feels broken no matter how quickly the food appears afterward.
LLM serving is that window. Time-to-first-token (TTFT) is the greeting: how long before the first word streams back. Tokens-per-second is how steadily the food comes out. Total latency is the whole order. Users forgive a long order far more than a long silence, which is why streaming exists. And the kitchen trick that makes the whole thing scale is batching: instead of cooking one order start-to-finish before touching the next, the kitchen interleaves — a new burger joins the grill the moment a spot opens, without waiting for the current batch of orders to all finish. That is continuous batching, and it is the single biggest reason modern LLM servers serve many users on one GPU.
Every serious deployment conversation you will have as an AI engineer or FDE lands here within ten minutes: "it feels slow," "the GPU bill is huge," "can we handle 200 concurrent users?" If you can decompose latency into TTFT vs generation rate, explain KV-cache memory pressure, and say when self-hosting on vLLM beats a hosted API (and when it absolutely does not), you sound like someone who has shipped. On Day 160's system-design interviews and on Day 161's production cutover, the latency budget you write today is the artifact you defend.
The drive-through window — why serving batches requests
step 1 / 5Five requests arrive at your LLM service within 200 ms. The GPU can process them one at a time… or together. This choice defines your latency AND your cost.
Guided practice
Measure TTFT vs total latency on a real endpoint
20 min- Create
latency_probe.pyfrom the starter code. Point it at your capstone's streaming endpoint (or any streaming LLM API you have a key for). - Run 20 requests with a short prompt and 20 with a long prompt (paste ~2 pages of text). Record TTFT and total time for each.
- Compute p50 and p95 for both metrics in both conditions. Notice: the long prompt moves TTFT (prefill work) far more than it moves tokens/sec.
- Now request a long OUTPUT ("write 500 words…"). Notice the opposite: TTFT barely moves, total time balloons. You have just separated prefill cost from decode cost empirically.
- Write four sentences: which metric moved, in which condition, and why.
Simulate static vs continuous batching
20 min- Run the starter simulation. It models a GPU that decodes one token per "tick" for every sequence in the batch (max batch 8), with requests of random output lengths arriving in a queue.
- Static mode: the batch is loaded, runs until EVERY member finishes, then reloads. Continuous mode: a finished sequence's slot is refilled from the queue on the very next tick.
- Compare the two printed numbers: mean wait time and total ticks to drain 60 requests. Continuous should win clearly — explain where the win comes from (idle slots).
- Set all output lengths equal and re-run. The gap nearly vanishes. Write one sentence on why variance in output length is what continuous batching exploits.
On your own
The latency budget worksheet
20 minWrite latency_budget.md for your capstone. Target: a user-perceived answer start under 1.5s at p95.
Break the request into stages — auth/routing, retrieval (embed query + vector search + rerank), prefill, decode — and assign each a measured or estimated p50 and p95 using today's probe data plus Day 142's traces. The stage budgets must sum to your target. Then name, for the single worst stage, one concrete optimization (smaller reranker? shorter context? streaming earlier?) and what it would cost you in quality.
Hints (only if stuck): retrieval is usually 100–400ms; prefill scales with context tokens; the biggest lever is almost always "send fewer tokens."
Capstone latency report
Produce docs/latency_report.md in the capstone repo: (1) a table of TTFT and total latency p50/p95 from at least 30 real requests against your staging deployment (Day 154), split by short vs long questions; (2) the stage-by-stage budget from practice; (3) one implemented improvement — e.g. start streaming before citations are assembled, cap max output tokens, or trim retrieved context — with before/after p95 numbers; (4) a five-line build-vs-rent paragraph: at your traffic, would self-hosting on vLLM ever pay? Show the arithmetic.
Common mistakes & misconceptions
- Reporting average latency. Production latencies are heavy-tailed (Day 58); p95 is what users feel and what SLOs (Day 157) are written against. Always report percentiles.
- Optimizing tokens/sec when the complaint is "it hangs before answering." That is TTFT — queueing plus prefill — and the fixes (streaming, shorter context, more capacity) are different.
- Believing generation is compute-bound. Decode is memory-bandwidth-bound; the KV cache, not FLOPs, caps concurrency. This is why quantization and paged KV memory help so much.
- Assuming self-hosting is cheaper because the per-token price disappears. A GPU billed 24/7 at low utilization costs far more per token than an API; do the utilization math first.
- Static-batching mental model: "batch of 8 means 8× throughput." The batch runs at the speed of its slowest member; continuous batching exists precisely to fix this.
- Testing latency only against localhost. Network, TLS, and cold starts belong in the number — measure against the deployed staging URL from Day 154.
Q1. Users say your assistant "freezes for 4 seconds, then the answer pours out fast." Which metric is broken, and which phase causes it?
Q2. Why does continuous batching beat static batching most when output lengths vary a lot?
Q3. The KV cache exists so that…
Go deeper — curated resources
- PagedAttention — the vLLM paper idea in the docs ↗ — KV cache managed as fixed-size blocks like virtual-memory pages; fragmentation stops capping batch size. Connects back to Day 39's virtual-memory mental model.
- Probe run against the deployed staging URL; percentiles recorded
- Batching simulation run in both modes with the variance experiment
- latency_report.md committed with a measured before/after improvement
- Quiz ≥ 2/3
← Back: Day 95 taught the per-token generation loop the KV cache accelerates; Day 107 gave you streaming and cost math; Day 151/154 gave you the deployed URL you measured today.
Forward →: Tomorrow (Day 156) attacks the cost side of the same numbers. Day 157 turns today's percentiles into SLOs, and Day 160's system-design answers lean on the quality/latency/cost reasoning you started here.
Unlocks: D156 Caching, Batching & Cost Engineering · D157 Monitoring & SLOs · D161 Week 23 Checkpoint: Production Cutover