Testing I — pytest Fundamentals
- Explain what automated tests buy you and when they pay for themselves
- Write pytest test functions that pytest discovers and runs automatically
- Structure every test as arrange–act–assert and name it as a behavior sentence
- Turn a table of cases into one parametrized test
- Run a single test, a single file, and a keyword-filtered subset from the terminal
| Spaced-rep warm-up: due flashcards | 10 min |
| ELI5 + tech read: discovery, AAA, parametrize | 20 min |
| Guided: first alarms + the parametrize upgrade | 35 min |
| Practice: bug hunt by test | 20 min |
| Project: grow the analyzer suite to 10+ tests | 25 min |
| Quiz + flashcards | 10 min |
Builds on: Day 15 — Clean, typed functions · Day 17 — The packaged analyzer + venv
Nobody installs a smoke alarm because they expect a fire tonight. They install it because they plan to live in the house for years, and over years, *something* will eventually smolder — usually in the one room nobody is watching. A test suite is a wall of smoke alarms for your code: each test watches one behavior ("parsing a valid line yields the right record"), and the moment any future change makes that behavior stop being true, the alarm goes off — within seconds, while the change is still fresh in your head and trivial to undo.
The seatbelt half of the analogy is about *when* the payoff comes. A seatbelt is mildly annoying every single day, and then one day it is the only thing that matters. Tests feel the same: writing assert parse_line("") is None today is unglamorous. But in week 12, when you refactor the parser at 11 p.m., you will change code with total confidence — because 40 alarms are standing guard over every behavior you ever promised. Untested code isn't code without bugs; it is code where nobody will ever *dare* to change anything.
Tests are the profession's dividing line: hobby code is verified by running it and squinting; professional code is verified by a suite anyone can run. Every later phase leans on this — the FastAPI service is tested with TestClient (Day 42), CI runs your suite on every push (Day 152), and the biggest idea in AI engineering, evals (Day 134), is literally "pytest for model behavior": a fixed set of cases, run after every change, guarding against regressions you can't see by squinting at one output. Learn the reflex on deterministic code first.
Guided practice
First alarms on the analyzer
20 min- Terminal: with your analyzer venv active,
pip install pytest, then create atests/folder at the project root with an emptytests/__init__.py(or nothing — pytest does not need it) andtests/test_parser.py. - Write the three tests from the starter code against YOUR
parse_line(adjust field names to match yours). Note each follows arrange–act–assert. - Terminal: run
pytest -qfrom the project root. Because Day 17 installed your package editable, the import just works — that is the src-layout payoff. - Break the parser on purpose (make it return status as a string), run
pytest -q, and READ the failure report: pytest shows both sides of the failed comparison. Revert the break. - Run a subset:
pytest -k malformed -qand then a single test by node id:pytest tests/test_parser.py::test_malformed_line_returns_none -q.
The parametrize upgrade
15 min- You are about to add a rule to the parser: status codes must be 100–599, else the line is malformed. Write the test FIRST as a parametrized table (starter code) — include valid edges (100, 599), invalid values (99, 600, 999), and a non-numeric case.
- Terminal: run
pytest -q— the new rows fail. Good: a failing test proves the test can fail, which is half its value. - Implement the range check in
parse_line, rerun, and watch the table go green row by row. - Count what you got: one test function, six cases, six independent pass/fail results. Add one more edge row of your own invention.
On your own
Bug hunt by test
20 minPaste this function into your project: it claims to return the top N most frequent items from a list, ties broken alphabetically. It contains two real bugs.
Your goal, in strict order: (1) write tests that EXPRESS the promised behavior (frequency descending, alphabetical ties, n larger than the number of distinct items); (2) run them and watch which fail; (3) fix the function until all pass. Do not fix by inspection first — let the tests find the bugs.
Hints: what order does sorted() use by default? How do you sort by two keys at once (hint: tuple keys, and negating a count reverses just that key)?
A real suite for the analyzer
Grow tests/test_parser.py into a suite of at least 10 tests covering the analyzer's pure core: line parsing (valid, malformed, empty, status-range rule via parametrize), the top-IPs counting logic, and the error-summary logic. Every test named as a behavior sentence, every test arrange–act–assert. Where a function is hard to test because it mixes file I/O with logic, refactor: split the pure logic out (Day 15 muscle) and test that — leave the I/O shell thin. Finish with pytest -q fully green and commit tests + refactors together. Tomorrow the suite learns to test the I/O shell too, using fixtures.
Common mistakes & misconceptions
- Testing only the happy path. The malformed line, the empty file, the tie in the ranking — edges are where bugs live and where tests earn their keep.
- Multiple unrelated asserts in one test. When it fails you can't tell which behavior broke; one behavior per test keeps alarms specific.
- Vague test names (test_1, test_parser_works). The name is the first thing you read on failure — make it state the promise that broke.
- Never watching a new test fail. A test that passed from birth might be asserting nothing; break the code (or write the test first) to prove the alarm has a battery.
- Copy-pasting near-identical tests instead of parametrizing. Ten copies hide the pattern and rot independently; one table shows intent and fails row by row.
- Writing tests that depend on each other or on run order. pytest may run any subset in any order; each test must arrange its own world.
Q1. Which function will pytest automatically collect and run?
Q2. Why write a failing test before fixing/implementing the behavior?
Q3. You have 8 input/expected pairs for one rule. The pytest-idiomatic structure is…
Go deeper — curated resources
- Evals are tests for models ↗ — Skim the opening: the eval loop you'll build on Day 134 is this exact discipline — fixed cases, run on every change — pointed at LLM behavior instead of parse_line.
- pytest -q green on a suite of ≥ 10 tests including a 5+ row parametrized table
- Both top_n bugs found by tests before reading the code for them
- One I/O-mixed function refactored for testability
- Quiz ≥ 2/3
← Back: Day 15's small pure functions are exactly what makes today painless — logic you can call with a value and assert on. Day 17's editable install is why tests/ can import your package cleanly.
Forward →: Day 19 adds fixtures, tmp_path, and mocking so the I/O shell gets tested too. Day 21 requires the suite to ship green. Day 141/152 put this suite (and eval suites — the same idea aimed at LLMs, Day 134) into CI where a red run blocks the merge.
Unlocks: D19 Testing II & Debugging · D21 Week 3 Checkpoint: Ship a Tested Package · D82 ML Code Structure & Pipelines