Day 47 · The pantry and the ticket rail

Caching & Queues

You will be able to

Name the cache layers a request can hit (client, CDN, app, DB) and what each is good for
Implement the cache-aside pattern with TTLs and write-time invalidation
Explain why cache invalidation is hard and choose between TTL, explicit invalidation, and versioned keys
Decouple slow work with a queue and workers, and explain backpressure
Debunk exactly-once delivery and state the real contract: at-least-once + idempotent consumers

Today's ~125 minutes

Spaced-rep warm-up: due flashcards (Days 43–46)	10 min
ELI5 + tech read + cache-flow visualizer	20 min
Guided: cache-aside lab + queue & backpressure	40 min
Practice: design the cache policy	20 min
Project: cache + worker upgrade	25 min
Quiz + flashcards	10 min

Builds on: Day 46 — Distributed systems — shared state & idempotency · Day 45 — Web service architecture & the rate-limiter dict · Day 40 — Concurrency — producers, consumers, races

The analogy

🧸 ELI5⚙ Tech

A busy kitchen survives on two tricks. The first is the pantry: instead of driving to the wholesale market for every onion (the database query), the cook grabs one from the pantry shelf (the cache). It is dramatically faster — but the pantry can lie. If the market changed suppliers this morning, the pantry onion is yesterday's truth. So the kitchen makes rules: throw pantry items out after a day (TTL), or, when the market calls with a change, walk over and clear that shelf (invalidation). Deciding which rule, for which shelf, is genuinely the hard part — feed a customer stale data and they notice.

The second trick is the ticket rail: when forty orders land at once, the waiter does not stand at the pass waiting for each dish. They clip tickets to the rail (enqueue) and go back to taking orders; cooks pull tickets when free (workers). If the rail fills up, the restaurant stops seating new tables for a moment (backpressure) instead of letting the kitchen drown. And because a ticket can fall and get re-clipped, cooks check the ticket number before firing a dish twice — the idempotency habit from yesterday, now applied to queues.

Why this matters on the job

LLM calls are the slowest, most expensive thing your services will ever do — seconds of latency, real dollars per request. Caching identical requests and queueing slow jobs are the two highest-leverage moves in AI cost engineering: Day 156 will cut your capstone's bill with exact and semantic caching, and every production ingest pipeline (Day 119's document indexing) runs as queued background work, not inside a request handler. Interviewers probe both: "where would you add a cache, and how does it go stale?" and "what happens when the queue backs up?" are standard system-design deep dives (Day 48 and Day 160).

Watch it happen

The pantry — a cache-aside read: miss, fill, hit

step 1 / 6

App→Cachethe pantry→Databasethe market

✉ GET user:42 → MISS

Cache-aside: the app ALWAYS asks the pantry (cache) before going to the market (database). First request for user:42 — the pantry is empty.

🎯 Predict modespeed1×

Guided practice

guided 1

Cache-aside from scratch — speed, staleness, invalidation

25 min

Run part 1 of the starter: a fake "database" query takes 200 ms. The cache-aside wrapper serves repeats from a dict with a TTL. Record the timings: first call vs the next ten.
Compute the hit rate and average latency for the 11-call sequence. This ratio is the whole business case for caching.
Part 2 demonstrates the lie: update the database directly, read again within TTL — you get the OLD value. Say out loud why: the cache has no idea the DB changed.
Fix it two ways: (a) call invalidate(key) inside the update function; (b) drop invalidation and shorten the TTL to 2 s. Note the trade-off in your notes — freshness by bookkeeping vs freshness by patience.
Kill the process and restart. The cache is empty (cold start) — every entry is a miss until it re-warms. This is why deploys briefly spike DB load.

🐍 python — editable, runs in your browser

import time

DB = {"note:1": "v1 of the note"}

def db_read(key):
    time.sleep(0.2)                    # the expensive trip to the market
    return DB[key]

cache = {}                             # key -> (value, expires_at)
TTL = 30.0

def get(key):
    hit = cache.get(key)
    if hit and hit[1] > time.time():
        return hit[0], "HIT"
    value = db_read(key)               # miss: go to the DB...
    cache[key] = (value, time.time() + TTL)   # ...and fill the cache
    return value, "MISS"

def invalidate(key):
    cache.pop(key, None)

# --- 1. the speedup
for i in range(11):
    t0 = time.perf_counter()
    value, status = get("note:1")
    ms = (time.perf_counter() - t0) * 1000
    print(f"call {i:2d}: {status:4s} {ms:6.1f} ms")

# --- 2. the lie: DB changes, cache does not notice
DB["note:1"] = "v2 EDITED"
print("after edit, cache serves:", get("note:1")[0])   # still v1!

# --- 3a. explicit invalidation fixes it
invalidate("note:1")
print("after invalidate:", get("note:1")[0])            # v2

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

A queue, two workers, and backpressure you can feel

15 min

Run the starter: a producer enqueues 20 jobs fast; two worker threads each take ~0.3 s per job. Watch jobs interleave across workers — this is horizontal scaling of work.
The queue is bounded at 5. Watch the producer log: once the queue fills, put() blocks until a worker frees a slot. That pause IS backpressure — the system self-throttles instead of buffering unboundedly.
Set the bound to 1000 and re-run: the producer finishes instantly, and all the waiting moves into the queue. Question for your notes: which behavior do you want when the producer is an HTTP handler with a customer waiting?
Simulate a crash: make a worker raise on job 7 without acking. Re-queue it in the except block and confirm another worker picks it up — at-least-once in eight lines.

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Design the cache policy

20 min

For your notes service, decide a caching policy for four data shapes: (a) GET /v1/notes list per user; (b) a single note body; (c) the API-key → user lookup that runs on EVERY request; (d) the rate-limit counters.

Your goal, per item: cache or not; where (in-process vs shared Redis-style); TTL length; invalidation strategy (TTL-only, explicit delete on write, or versioned key); and one sentence on what staleness costs if you get it wrong.

Hints (only if stuck): (c) is hot, small, and must reflect key revocation fast — short TTL or explicit invalidation on revoke; (d) is not a cache at all — it is shared STATE and must not be evictable the same way; losing it resets limits.

Ship before you stop

Cache + worker upgrade for the notes service

Two upgrades, committed with tests. (1) Cache-aside on GET /v1/notes/{id}: an in-process TTL cache module (dict-based, like the guided lab — note in a docstring why multi-replica needs Redis, per Day 46), with explicit invalidation in the update and delete paths. Tests: a repeated read hits the cache (count db calls with a spy repo); an update then read returns the NEW body. (2) A background export worker: POST /v1/notes/{id}/export enqueues a job (bounded queue.Queue, worker thread writes a text file to an exports dir) and returns 202 with a job id; jobs table or dict tracks status; the worker is idempotent — re-delivering the same job id does not produce a duplicate file. Finish caching_notes.md with your practice-policy table plus three sentences on semantic caching for LLM responses (preview of Day 156).

Rubric — check what you completed (0/6)

Spy-repo test proves the second read skips the databaseUpdate path invalidates: read-after-write returns fresh dataExport endpoint returns 202 immediately; work happens on the worker threadRe-enqueueing a completed job id is a no-op (idempotent consumer)Policy table covers all four data shapes with TTL + invalidation choices justifiedCommitted with a descriptive message

Common mistakes & misconceptions

Caching without an invalidation story. "We will add a TTL later" means serving stale data in prod. Decide staleness tolerance per key BEFORE caching, not after the bug report.
Confusing a cache with a store. Caches are evictable by design — anything you cannot afford to lose (rate counters, sessions, job state) needs a store with persistence guarantees, even if the same tool (Redis) can play both roles.
Caching inside a single process and calling it shared. Two replicas, two dicts, two truths — Day 46 again. Shared cache or per-replica staleness, choose knowingly.
Unbounded queues "so producers never block." The queue becomes a memory leak with a delay; the system fails later and worse. Bound it and let backpressure do its job.
Believing a framework that advertises exactly-once delivery. Read the fine print: it is at-least-once delivery plus deduplication (idempotency) at the consumer. You still have to build the consumer side.
Running slow work in the request handler "because it is only 2 seconds." At 50 concurrent users that is a pile-up of blocked connections. Enqueue, return 202, let workers grind.

Knowledge check

Q1. In cache-aside, what happens on a cache miss?

Q2. Why is exactly-once DELIVERY considered a myth over a network?

Q3. Your bounded job queue is full and the producer blocks. What is this behavior called, and is it a bug?

Go deeper — curated resources

repoSystem Design Primer — caching & asynchronism sections ↗30 min articleByteByteGo — caching strategies & message queue explainers ↗20 min repoDDIA references — ch. 8 (trouble with distributed systems) & ch. 11 (stream processing) ↗15 min courseCMU 15-445 — buffer pools: the database's own cache ↗15 min

If you have a third hour

Thundering herd & cache stampedes — When a hot key expires, hundreds of requests miss simultaneously and hammer the DB. Mitigations: per-key locks, probabilistic early refresh, jittered TTLs. You will meet this in production someday — recognize it.

Done means

Cache-aside lab run: hit-rate speedup measured, staleness demonstrated and fixed both ways
Backpressure observed with the bounded queue; crash-requeue experiment done
Cache policy table complete for all four data shapes
Notes service upgrade committed with passing spy-repo and idempotent-worker tests
Quiz ≥ 2/3

How this connects

← Back: Day 46 is everywhere today: the cache is shared state done deliberately, at-least-once redelivery is the ambiguous timeout, and idempotent consumers are idempotency keys wearing a hard hat. The producer/consumer threads are Day 40's concurrency, now with a purpose.

Forward →: Tomorrow's design method (Day 48) treats "add a cache" and "add a queue" as standard moves you can now justify, and Day 49's URL shortener uses cache-aside for hot links. Day 122 caches conversation context, and Day 156 brings the big payoff: exact + semantic caching to cut LLM cost, with prompt-prefix caching as the provider-side version.

Unlocks: D48 System Design Method · D49 Week 7 Checkpoint: Design & Build a URL Shortener · D107 LLM APIs II — Streaming, Retries & Cost · D149 Compose, Registries & Image Hygiene