Iterators & Generators
- Explain the iteration protocol — iter(), next(), StopIteration — and desugar a for loop by hand
- Write generator functions with yield and predict exactly when their code runs
- Replace list-building loops with generator expressions where laziness pays
- Chain generators into a streaming pipeline that processes a large file in constant memory
- Use itertools (islice, count, chain) to manipulate streams without materializing them
| Spaced-rep warm-up: due cards | 10 min |
| Concept study: ELI5 + tech — desugar a for loop on paper | 20 min |
| Guided: protocol by hand, freeze-frame, pipeline build | 45 min |
| Practice: the memory showdown | 20 min |
| Project: streamstats | 15 min |
| Quiz + flashcards | 10 min |
Builds on: Day 4 — Collections and iteration · Day 5 — Reading files line by line · Day 9 — Dunder methods (the protocol)
A warehouse approach to making a million sandwiches: make all million first, stack them to the ceiling, then start serving. You'll need a warehouse (memory), and the first customer waits until the last sandwich is made. The conveyor belt approach: make one sandwich when a customer actually asks, hand it over, make the next on the next request. No warehouse. First customer served instantly. And if only ten customers show up, you made ten sandwiches, not a million.
A Python list is the warehouse: every element exists in memory at once. A generator is the conveyor belt: a function with yield that produces one item on demand, then *freezes mid-sentence* — variables intact — until asked again. Nothing runs until someone asks; asking is next(); and a for loop is just polite, repeated asking. Belts also snap together: one belt reads lines from a huge file, the next keeps only errors, the next extracts a field — and one item flows through the whole chain at a time. A 10 GB file streams through a pipeline like that using a few kilobytes of memory. That's the trick warehouses can never do.
Streaming is a production survival skill: "load it all into memory" is exactly how services die at 3am when a customer's export is 100x your test file — the OOM killer (Day 39) has no mercy. You've already relied on this material: for-line-in-f (Day 5) works because files are iterators. It's also the native shape of AI engineering: LLM responses arrive as token streams (Day 107), Dataset/DataLoader pipelines feed training loops one batch at a time (Day 88), and RAG ingestion (Day 113) is read-chunk-embed as a stream because corpora don't fit in RAM. Interviewers probe it with one cheap question: "what does yield do?"
Guided practice
Pull back the curtain — the protocol by hand
15 min- Create
week-02/protocol_lab.pywith the starter code. Section 1: drive a list's iterator manually with iter/next until StopIteration fires. You are being the for loop. - Confirm the two-loops mystery: loop the LIST twice (works — fresh iterator each time), then loop the same ITERATOR twice (second loop gets nothing). Write the rule as a comment.
- Section 2: files are iterators. Open your Day 6 app.log and pull exactly three lines with next(f). Close and reopen to reset — there is no rewind button on a belt.
- Section 4: your OWN class joins the protocol — a Countdown whose __iter__ returns a generator. Loop over an instance of it. Day 9's dunder story completes: for works on anything that plays the protocol.
- In one sentence: what does a for loop do when next() raises StopIteration? (It catches it and ends cleanly — the exception is the protocol's "belt is empty" signal, not an error.)
Watch it freeze — generator mechanics
15 min- Create
gen_lab.pywith the starter code — a generator with loud print statements so you can SEE the pausing. Before running, write down the exact order of output you expect. - Run it. The headline observations: "STARTING" prints only at the first next(), not at the call; between yields the function is frozen; "FINISHED" prints during the final, StopIteration-raising next. If any of the three surprised you, run it again stepping through in your head.
- Prove locals survive the freeze: add a running total variable inside the generator that accumulates across yields, and yield it at the end.
- Convert: rewrite squares_list (builds a warehouse) as a generator expression, and get the sum of the first 1000 squares without a list: sum(n * n for n in range(1000)).
- Meet infinity safely: from itertools import count, islice; take the first 5 from naturals = count(1). Then try list(count(1)) — actually, DON'T; write a comment explaining what would happen and why islice makes infinite belts usable.
Build the belt line — a streaming log pipeline
15 min- First, manufacture a BIG log: adapt Day 6's make_log.py to write 200_000 lines (bump the range). Note the file size with ls -l.
- Create
pipeline_lab.pywith the starter code — three chained generator stages ending in a frequency count. Run it and note: results appear fast, and memory stays flat no matter the file size. - Prove the laziness: add a print("parsing!") inside the parse stage, then run only the first three lines of the pipeline (build the generators, iterate nothing). Silence. Then pull one item with next(). One "parsing!". Demand drives the belt.
- Compare with the warehouse: write the same logic with lists at each stage (read all lines, filter all, split all). On 200k lines both may finish — now imagine 200 GB and write one sentence on which version survives (and connect it to for-line-in-f from Day 5).
- Extend the pipeline with one more stage: keep only errors from IPs starting "10." and report the top offender. One new generator line, no other changes — that pluggability is the payoff of belts.
On your own
The memory showdown
20 minSettle warehouse-vs-belt with numbers. Build showdown.py.
Goal: compute the sum of squares of the first 10 million integers two ways — (a) build the full list then sum it; (b) sum a generator expression. Measure both: wall time with time.perf_counter (Day 22 formalizes this harness), and peak memory with tracemalloc (start it, run one version, take tracemalloc.get_traced_memory(), reset between runs). Print a two-row comparison table and write the conclusion as a comment: how many times more memory did the warehouse cost, and was it faster or slower?
Constraints: measure the two versions in separate runs or reset tracemalloc carefully between them; report peak (the second number), in MB.
Hints (only if stuck): import tracemalloc; tracemalloc.start(); ...; current, peak = tracemalloc.get_traced_memory(); tracemalloc.stop(). Expect the list to cost hundreds of MB; the generator, kilobytes.
streamstats — constant-memory file statistics
Build streamstats.py, a tool that reports statistics for a numbers file of ANY size: count, min, max, mean — in one pass, in constant memory, skipping junk lines (Day 5's resilient-summer rules). Structure it as belts: a generator reading raw lines, a generator that yields successfully-parsed floats (EAFP inside), and a consumer loop that updates running count/min/max/total — no list of values may ever exist. Generate a test file with 1 million values (plus sprinkled junk) to prove it, and verify against sum()/len() on a small file where a warehouse is checkable. Stretch: use islice to add a --preview mode showing the first 5 parsed values. Commit it — Day 14's log analyzer will reuse the read-parse-consume skeleton verbatim.
Common mistakes & misconceptions
- Expecting the body to run when you call a generator function. Calling builds the belt; only next() (or a for loop) runs code. Print-debug inside one and silence means nobody has pulled yet.
- Iterating a generator twice. It exhausts; the second loop silently gets nothing — no error, just wrong results. Recreate it, or materialize ONCE with list() if you truly need two passes.
- Calling len() or indexing on a generator. Streams have no length or random access. islice for slicing; count-as-you-go for length; or admit you need a list.
- Materializing by reflex: list(gen) then a loop. You just bought the warehouse to use a belt. Feed generators directly to for, sum, max, any.
- Using a genexp when you need the data twice (checking "if x in gen" then looping gen). First use consumed part of it. Single-pass means SINGLE pass.
- Returning a value with return inside a generator expecting callers to see it. return in a generator just stops the belt (it rides inside StopIteration). Yield results; don't return them.
Q1. g = countdown(3) — the generator function with yield. What has executed?
Q2. Processing a 50 GB log on a 16 GB machine. Which works?
Q3. evens = (n for n in nums if n % 2 == 0); print(list(evens)); print(list(evens)). Second print?
Go deeper — curated resources
- yield as a two-way street — Generators can also RECEIVE values (gen.send(x)) and delegate (yield from) — the machinery Python's async/await (Day 40) was originally built on. Skim the "yield expressions" reference once so Day 40 feels like a reunion.
- Protocol lab done: manual next() drill, exhaustion rule written, custom iterable class works
- Noisy-generator output order predicted correctly (or the miss explained)
- Pipeline processes the 200k-line log with a stage added painlessly
- Showdown table produced with real memory numbers and a written conclusion
- streamstats committed and verified against the small file; quiz ≥ 2/3
← Back: for-line-in-f from Day 5 was this lesson in disguise — files are iterators. The pipeline architecture is Day 6's shell pipes rebuilt in Python, and __iter__ completes the dunder-protocol story Day 9 started.
Forward →: Day 12 wraps functions around functions the way today chained belts. The streaming mindset returns with money attached: Day 40's async streams, Day 88's DataLoaders, Day 107's token-by-token LLM responses, and Day 113's RAG ingestion are all conveyor belts. Day 22 prices the warehouse in Big O terms.
Unlocks: D12 Closures, Decorators & Functional Style · D13 Regex & Text Processing · D14 Week 2 Checkpoint: Log Analyzer · D22 Big O & Complexity