Day 40 · One chef, many pots

Concurrency & Async Python

You will be able to

Distinguish concurrency from parallelism and I/O-bound from CPU-bound work
Explain what the GIL does and does not prevent
Choose correctly among threads, processes, and asyncio for a given workload
Write async/await code with gather, timeouts, and cancellation
Demonstrate a race condition and fix it with a lock

Today's ~120 minutes

Spaced-rep warm-up: due cards (processes, signals, fds)	10 min
ELI5 + tech read; watch the async-loop visualizer park and resume tasks	18 min
Guided: workload benchmark, asyncio toolkit + the race	42 min
Practice: the polite crawler	20 min
Project: benchmark CLI + decision card	20 min
Quiz + flashcards	10 min

Builds on: Day 39 — OS essentials — processes & threads · Day 11 — Iterators & generators · Day 12 — Closures & decorators

The analogy

🧸 ELI5⚙ Tech

Watch a good cook make a four-dish dinner. They do NOT cook dish one to completion, then start dish two — they put the rice on, and WHILE it simmers, chop vegetables; while the sauce reduces, sear the fish. One cook, many pots, zero waiting around. That is concurrency: making progress on many tasks by using the WAITING time of some to work on others. Parallelism is different — that is hiring three more cooks (more CPU cores) so four things literally happen at the same instant.

Which one you need depends on what is slow. Simmering (waiting on the network, the disk, a database, an LLM API) needs the one-clever-cook trick — most of the elapsed time is waiting, so one worker who never idles can juggle hundreds of pots. Nonstop chopping (pure computation) gains nothing from juggling — one pair of hands is busy every second — so only more cooks help.

Python's kitchen has a famous rule: the GIL, one "knife" that threads must hold to run Python code. So threads take turns chopping — fine for simmering-heavy work (waiting threads give up the knife), useless for chop-heavy work. Asyncio removes even the pretense of multiple cooks: one cook, an explicit to-do list (the event loop), and await marking every spot where they switch pots.

Why this matters on the job

LLM applications are the most I/O-bound software mainstream engineering has ever produced: every call to a model API is seconds of pure waiting, and an app that embeds 500 chunks (Day 115) or fans one question out to three models sequentially is unusably slow — while the async version finishes in roughly the time of the slowest single call. This is why FastAPI (tomorrow) is async-native, why every serious LLM SDK ships an async client (Day 106), and why your capstone's ingest pipeline lives or dies by a semaphore. Interviewers, meanwhile, love "explain the GIL" precisely because it separates people who can parrot "Python can't do threads" from people who know threads are exactly right for I/O.

Watch it happen

One chef, many pots — the event loop's ready queue

step 1 / 6

front

A: call LLM API

B: parse doc

C: save row

back

One thread, three tasks. The event loop is a chef working a queue: run the front task until it finishes — or until it must WAIT for something.

🎯 Predict modespeed1×

Guided practice

guided 1

The workload decides: threads vs processes vs the GIL

20 min

Create concurrency_lab.py with the starter: an I/O-bound task (sleep 0.5s, simulating an API call) and a CPU-bound task (summing 10 million numbers).
Predict BEFORE running: 8 I/O tasks sequentially vs with 8 threads; 4 CPU tasks sequentially vs with 4 threads vs with 4 processes. Write your predicted times down.
Run the harness. Expected shape: I/O with threads collapses ~4s to ~0.5s (waiting overlaps); CPU with threads is NO faster (the GIL); CPU with processes divides by your core count.
Explain each row of the printed table out loud using the GIL model: who was holding the knife, who was waiting, who had their own kitchen.
Note the pattern in the code: ThreadPoolExecutor and ProcessPoolExecutor share one API (executor.map) — the decision is one class name, so the THINKING is the deliverable.

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

asyncio for real + the race you must see once

22 min

In async_lab.py, write fetch(name, seconds) as an async coroutine (await asyncio.sleep to simulate an LLM API call). Run three sequentially with plain awaits, then concurrently with gather — 3.0s becomes ~1.2s (the slowest call). This is the shape of every multi-model / multi-chunk call you will ever make.
Add the cardinal-sin demo: replace one task's await asyncio.sleep with blocking time.sleep and watch EVERYTHING serialize — one blocked cook, whole kitchen frozen. Restore it and say the rule: never block the loop.
Add wait_for with a 1.0s timeout around a 5s task: TimeoutError fires, and the task's except CancelledError block runs its cleanup. Cancellation is cooperative — it arrives at the next await.
Add the semaphore: 10 tasks, asyncio.Semaphore(3), and a live printout of in-flight count — never above 3. This is your future rate-limit governor, verbatim.
The race. Run the threaded counter in the starter: two threads each do 500,000 unguarded count += 1; the total comes up SHORT (run it thrice — different shortfalls, the signature of a race). Add the Lock and get exactly 1,000,000. State why the GIL did not save you: += is several bytecodes, and preemption lands between them.

🐍 python — editable, runs in your browser

import asyncio, threading, time

async def fetch(name, seconds):
    print(f"  -> {name} started")
    try:
        await asyncio.sleep(seconds)          # the polite, loop-friendly wait
    except asyncio.CancelledError:
        print(f"  !! {name} cancelled -- cleaning up")
        raise
    print(f"  <- {name} done")
    return name

async def main():
    t0 = time.perf_counter()
    await fetch("a", 1.0); await fetch("b", 1.2); await fetch("c", 0.8)
    print(f"sequential: {time.perf_counter() - t0:.2f}s")

t0 = time.perf_counter()
    results = await asyncio.gather(fetch("a", 1.0), fetch("b", 1.2), fetch("c", 0.8))
    print(f"gather:     {time.perf_counter() - t0:.2f}s -> {results}")

try:
        await asyncio.wait_for(fetch("slowpoke", 5.0), timeout=1.0)
    except asyncio.TimeoutError:
        print("timed out -- slowpoke was cancelled")

sem = asyncio.Semaphore(3)                 # the rate-limit governor
    in_flight = 0
    async def governed(i):
        nonlocal in_flight
        async with sem:
            in_flight += 1
            print(f"  task {i} running ({in_flight} in flight)")
            await asyncio.sleep(0.5)
            in_flight -= 1
    await asyncio.gather(*(governed(i) for i in range(10)))

asyncio.run(main())

# ---- the race ----
count = 0
lock = threading.Lock()
USE_LOCK = False                               # flip after observing the loss

def bump():
    global count
    for _ in range(500_000):
        if USE_LOCK:
            with lock:
                count += 1
        else:
            count += 1                         # read-modify-write: interleavable

threads = [threading.Thread(target=bump) for _ in range(2)]
for t in threads: t.start()
for t in threads: t.join()
print(f"count = {count:,} (expected 1,000,000)")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

The polite crawler

20 min

Build crawler.py: given 20 simulated "URLs" (each fetch is asyncio.sleep(random.uniform(0.2, 1.5)) returning fake content, with a 10% chance of raising a fake ConnectionError), fetch them all with: at most 4 in flight (semaphore), a 1.0s per-fetch timeout, ONE retry with a short backoff for failures/timeouts, and a final report — successes, failures, total elapsed vs the sum of individual times (your concurrency win).

Constraints: no fetch may block the loop; gather must not die because one URL failed (look up gather's return_exceptions=True, or wrap each fetch). Test that in-flight never exceeds 4 by printing a counter.

Hints: structure per-URL logic as its own coroutine (governed fetch → timeout → retry) and gather those. random.random() < 0.1 triggers the fake failure.

Ship before you stop

The concurrency decision card + a benchmark you own

Two deliverables. (1) bench_concurrency.py: extend the guided harness into a clean, argparse-driven benchmark (--workload io|cpu --strategy seq|threads|processes|async --n 8) with logging (Day 16) and a results table; include asyncio for I/O workloads and note honestly why async has no CPU row. (2) concurrency_card.md: your decision card — the workload-diagnosis question, the three tools with one-line verdicts each, the GIL in two sentences YOU wrote, the cardinal sin of async, and the race-condition rule. This card is the reference you will consult on Day 107 (streaming clients), Day 115 (parallel embedding), and Day 155 (serving concurrency). Commit both.

Rubric — check what you completed (0/5)

Benchmark runs all valid workload×strategy combos from the CLI and prints a comparison tableI/O results show threads and async both collapsing wait time; CPU results show only processes scalingRace demo included with lock fix and a comment locating the interleaving pointDecision card states the GIL's effect on I/O vs CPU threads correctly and in your own wordsCommitted; card linked from the repo README

Common mistakes & misconceptions

Saying "Python threads are useless because of the GIL." Blocked I/O releases the GIL — threads are exactly right for I/O-bound work. The GIL nullifies CPU threading only.
Believing the GIL makes code thread-safe. count += 1 is multiple bytecodes; threads interleave between them and drop updates. Shared mutation needs a Lock regardless.
Calling time.sleep or requests.get inside async code. One blocking call freezes every task on the loop — use await asyncio.sleep, an async client, or to_thread.
Writing async def everywhere and awaiting each call in sequence — that is sequential code with extra keywords. The win comes from gather / TaskGroup running coroutines together.
Unbounded gather over 5,000 URLs: you just DDoSed the API and blew through rate limits. A Semaphore capping in-flight work is not optional in production.
Using multiprocessing for I/O-bound work: processes cost real memory and startup, and picklable-argument restrictions bite — threads or async do it cheaper.

Knowledge check

Q1. Four CPU-heavy parsing jobs run in 4 threads and take the same time as sequential. Why, and what fixes it?

Q2. One task in your asyncio app calls time.sleep(5). What happens?

Q3. Two threads each increment a shared counter 500k times without a lock; the result is 861,204. What happened?

Go deeper — curated resources

docsasyncio — official Python docs (coroutines & tasks) ↗30 min docsthreading — official Python docs (Lock, and the GIL notes) ↗15 min bookOSTEP — Concurrency chapters (threads, locks) ↗25 min

If you have a third hour

Structured concurrency: asyncio.TaskGroup — Python 3.11's TaskGroup scopes tasks to a block and propagates failures cleanly — the modern alternative to bare gather. Rewrite the polite crawler with it and compare error behavior.

Done means

Benchmark table produced; every row explained via the GIL model
gather, wait_for + cancellation cleanup, and semaphore all demonstrated
Race observed (short count), then fixed exactly with a Lock
Polite crawler respects max-4 in flight with timeout + retry; card committed
Quiz ≥ 2/3

How this connects

← Back: Threads and processes are Day 39's roommates-vs-rooms, now raced; coroutines are Day 11's pausable generators grown up; the with-lock pattern reuses Day 5's context-manager discipline.

Forward →: Tomorrow FastAPI runs your handlers on this exact event loop — the cardinal sin becomes a production outage. Day 107 streams LLM responses async, Day 115 embeds corpora under a semaphore, and Day 155 tunes serving concurrency with today's vocabulary.

Unlocks: D41 HTTP & Build Your First API · D42 Week 6 Checkpoint: API + DB Mini-Service · D46 Distributed Systems Fundamentals · D47 Caching & Queues