Day 156 · The pantry, again — at scale

Caching, Batching & Cost Engineering

You will be able to

Implement exact and semantic response caching and measure hit rate vs staleness risk
Explain prompt-prefix caching and structure prompts to exploit it
Control output length deliberately, since output tokens usually cost several times input tokens
Route requests between a cheap and an expensive model by difficulty, with an escalation rule
Account cost per request and per feature, and cut your capstone's cost measurably

Today's ~120 minutes

Spaced-rep warm-up: due cards incl. Day 47 caching & Day 129 routing	10 min
ELI5 + tech read; list the four levers from memory	15 min
Guided: two-tier cache + cost ledger	40 min
Practice: difficulty router with escalation	20 min
Project: cost report with before/after and projections	25 min
Quiz + flashcards	10 min

Builds on: Day 47 — Caching & queues — cache-aside, TTL, invalidation · Day 107 — LLM APIs II — cost math per feature · Day 129 — Model selection — cascades & routers · Day 155 — Serving & inference optimization

The analogy

🧸 ELI5⚙ Tech

On Day 47 the pantry was a metaphor for keeping popular ingredients within arm's reach. Now imagine the restaurant discovers that 30% of orders are literally the same dish — "the usual" — and starts plating those from a warming shelf instead of cooking from scratch. That is response caching. Then the chef notices something subtler: hundreds of different orders all start with the same base — the same stock, the same sauce. So the kitchen pre-makes the base once and only cooks the part that differs. That is prompt-prefix caching: your system prompt and few-shot examples are the stock; the provider charges you a fraction to reuse the already-processed prefix.

The last trick is staffing. You do not put the executive chef on grilled-cheese orders. A router looks at each ticket and sends easy ones to the line cook (small, cheap model) and genuinely hard ones to the executive chef (frontier model) — with a rule that a line-cook failure escalates rather than ships. Caching, prefix reuse, shorter outputs, and routing are the four levers that routinely cut LLM bills by half or more without touching quality where it matters.

Why this matters on the job

Cost is where AI projects die quietly. A demo that costs $0.04/request is charming at 100 requests/day and a budget crisis at 100k. FDEs get asked "what will this cost at scale?" in the first customer meeting, and "why did the bill triple?" in month two. Engineers who can show a cost-per-request breakdown and then cut it 40% with caching and routing — while proving quality held via the Day 140 eval harness — are the ones trusted with production. Day 173 turns this skill into customer-facing ROI math.

Watch it happen

The pantry — a cache-aside read: miss, fill, hit

step 1 / 6

App→Cachethe pantry→Databasethe market

✉ GET user:42 → MISS

Cache-aside: the app ALWAYS asks the pantry (cache) before going to the market (database). First request for user:42 — the pantry is empty.

🎯 Predict modespeed1×

Guided practice

guided 1

Build a two-tier cache wrapper

25 min

Create llm_cache.py from the starter. Tier 1 is an exact cache keyed on a hash of (model, normalized prompt, params). Tier 2 is a semantic cache using sentence-transformers embeddings with a cosine threshold.
Wire it in front of your capstone's answer function (cache-aside, Day 47 pattern): check exact → check semantic → call model → store.
Replay 30 questions from your Day 140 golden set, then replay 15 paraphrases of them (write them or generate them). Record: exact hits, semantic hits, misses.
Lower the semantic threshold from 0.92 to 0.80 and replay again. Find a paraphrase pair where the cache now returns a WRONG answer — that is the staleness/precision trade made visible.
Write down your chosen threshold and TTL, with one sentence of justification each.

🐍 python — editable, runs in your browser

import hashlib, json, time
from sentence_transformers import SentenceTransformer
import numpy as np

_embedder = SentenceTransformer("all-MiniLM-L6-v2")

class LLMCache:
    def __init__(self, sim_threshold=0.92, ttl_s=3600):
        self.exact, self.entries = {}, []   # entries: (vec, answer, ts)
        self.t, self.ttl = sim_threshold, ttl_s

def _key(self, model, prompt, params):
        blob = json.dumps({"m": model, "p": prompt.strip().lower(), "x": params},
                          sort_keys=True)
        return hashlib.sha256(blob.encode()).hexdigest()

def get(self, model, prompt, params):
        k = self._key(model, prompt, params)
        if k in self.exact and time.time() - self.exact[k][1] < self.ttl:
            return self.exact[k][0], "exact"
        v = _embedder.encode(prompt, normalize_embeddings=True)
        for vec, ans, ts in self.entries:
            if time.time() - ts < self.ttl and float(np.dot(v, vec)) >= self.t:
                return ans, "semantic"
        return None, "miss"

def put(self, model, prompt, params, answer):
        self.exact[self._key(model, prompt, params)] = (answer, time.time())
        v = _embedder.encode(prompt, normalize_embeddings=True)
        self.entries.append((v, answer, time.time()))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Cost accounting per request

15 min

Add a cost_ledger to your capstone: after every model call, log a structured line with feature, model, input_tokens, output_tokens, cache_status, and computed dollars (put your provider's real prices in a PRICES dict).
Replay your golden set through the app once with caching off, once with caching on.
Run the starter aggregation script over the log: cost per request p50/p95, cost per feature, and the percentage saved by caching.
Sanity-check against the provider dashboard's usage page — if your ledger disagrees by more than ~10%, find the uncounted call (retries and judge calls are the usual suspects).

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

The router that must not lie

20 min

Add difficulty routing to the capstone: questions matching "easy" criteria (short, single-fact, high retrieval confidence) go to the small model; everything else to the big one. Your escalation rule: if the small model's answer fails citation validation or the groundedness check, retry on the big model — never ship the failure.

Constraints: (1) re-run the Day 140 golden set and report pass rate per route — overall quality must not drop more than 2 points; (2) report the cost saving; (3) log the route decision so Day 157's dashboard can chart route mix.

Hints: retrieval score percentiles from Day 117 make a decent difficulty signal; when in doubt, route up.

Ship before you stop

Cut the capstone's cost, with receipts

Produce docs/cost_report.md: (1) baseline cost per request and per 1k requests from the ledger, broken down by feature; (2) the three levers you applied — caching, prompt reordering for prefix cache, output caps, or routing (pick at least two) — each with its measured saving; (3) the quality guardrail: golden-set pass rate before and after, with the Day 139 error bars; (4) a projection table: monthly cost at 1k, 10k, and 100k requests/day, before vs after. This document feeds directly into Day 173's ROI one-pager.

Rubric — check what you completed (0/5)

Ledger reconciles with provider dashboard within 10%At least two levers implemented with individually measured savingsCombined saving ≥ 25% on the replayed golden-set workloadGolden-set pass rate after changes within 2 points of baseline, CIs shownProjection table present with arithmetic a stakeholder can check

Common mistakes & misconceptions

Shipping a semantic cache without an eval. A 0.85-similarity hit on a different-jurisdiction question is a confident wrong answer; tune the threshold against labeled paraphrase pairs, not vibes.
Forgetting to invalidate the response cache when the RAG corpus updates. The retrieval index changed but the cache still serves answers grounded in deleted documents.
Putting volatile content (retrieved chunks, timestamps, user name) at the TOP of the prompt — it breaks the shared prefix and forfeits prefix-cache discounts. Stable first, volatile last.
Optimizing input tokens while ignoring output. Output tokens usually cost 3–5× more; a max-token cap and a brevity contract often save more than any retrieval trim.
Routing to the cheap model with no escalation path. The rule is "cheap first, but never ship a validation failure" — otherwise your cost cut is a silent quality cut.
Measuring savings on cache-warm replays only. Report steady-state hit rate on realistic traffic, not the 100% hit rate of replaying the same 30 questions twice.

Knowledge check

Q1. Your prompt is: [retrieved chunks][system instructions][few-shot examples][question]. Why is prefix caching saving you almost nothing?

Q2. A semantic cache at threshold 0.80 returns the France leave-policy answer for a Germany question. The correct first fix is…

Q3. Why must the golden set be re-run after adding a cheap-model route?

Go deeper — curated resources

docsClaude Docs — prompt engineering (structure prompts for caching) ↗20 min repoOpenAI Cookbook — caching & cost patterns ↗20 min articleChip Huyen — blog (cost & latency engineering posts) ↗20 min docsSentence-Transformers docs — embeddings for the semantic cache ↗15 min

If you have a third hour

Request coalescing — When 50 identical requests arrive in the same second (a shared dashboard, a retry storm), let one call the model and 49 await its result. Same idea as the thundering-herd cache guard from Day 47.

Done means

Cache wrapper live in the capstone with a justified threshold and TTL
Cost ledger reconciles with the provider dashboard
cost_report.md committed: ≥25% saving, quality within 2 points, projections included
Quiz ≥ 2/3

How this connects

← Back: This is Day 47's cache-aside pattern and TTL discipline applied to LLM responses, Day 107's cost math made operational, and Day 129's routing decision finally implemented with an eval guardrail from Day 140.

Forward →: Day 157 charts cost/request on the monitoring dashboard. Day 160's design answers must name these levers unprompted, and Day 173 converts today's cost report into the customer-facing ROI one-pager.

Unlocks: D173 ROI, Pricing & Cost Analysis