Regex & Text Processing
- Read and write patterns using literals, character classes, quantifiers, groups, and anchors
- Choose correctly among re.search, re.findall, re.finditer, and re.sub
- Extract structured fields from log lines using named groups
- Explain greedy vs lazy matching and fix a pattern that over-matches
- Judge when regex is the wrong tool and use string methods or a parser instead
| Spaced-rep warm-up: due cards | 10 min |
| Concept study: ELI5 + tech — say five patterns aloud in English | 20 min |
| Guided: shapes, groups, greed & verbose | 45 min |
| Practice: the extraction gauntlet | 20 min |
| Project: logparse.py for tomorrow | 15 min |
| Quiz + flashcards | 10 min |
Builds on: Day 5 — String methods and text files · Day 11 — Streaming lines (today's patterns plug into those pipelines)
Your editor's find box matches exact text: search "cat" and you get "cat". A regular expression is a find box that speaks in *shapes*: "find me a 4-digit number", "find anything shaped like an email address", "find ERROR, but only at the start of a line". You describe the shape once — \d means "any digit", + means "one or more of the previous thing", so \d+ is "a run of digits" — and the regex engine hunts every match in a haystack of any size.
Think of it as describing a suspect to a sketch artist instead of showing a photo. A photo ("cat") matches one face. A description ("digits, then a dash, then digits") matches every phone number, invoice ID, or date range that fits the shape. Groups — parentheses — add a second superpower: capture the parts you care about. "Match date-space-level-space-message, and hand me the three pieces separately" turns a raw log line into structured data, one line of code. The catch: regex is a language of pure punctuation, write-only if you're careless. Today you learn to read it, write it politely, and — just as important — recognize the jobs it should refuse.
Text is the substrate of AI engineering: cleaning documents before chunking (Day 114), scrubbing PII with redaction patterns (Day 132's guardrails are regex at the front line), extracting fields from semi-structured LLM output when JSON mode isn't available, and parsing logs during incidents (Day 172 — the customer's log format is always slightly weird, and the FDE who can regex it live looks like a wizard). Tomorrow's checkpoint project runs on today's named groups. And "write a pattern to match X" still appears in screening interviews because it cheaply reveals who has processed real text.
Guided practice
Shapes, not photos — first patterns
15 min- Create
week-02/regex_lab.pywith the starter code. Run section 1 and match each result to its pattern. For each of the five patterns, say the shape ALOUD in English before looking at the output ("one or more digits", "the word ERROR at the start of a line"...). Verbalizing patterns is the actual skill. - Predict-then-run: before section 2 executes, write what findall will return for each call as a comment. The \berror\b vs error distinction ("terrors"!) is the one to get right.
- Modify: change the year pattern to match BOTH 2-digit and 4-digit years (quantifier range {2,4}).
- Add one pattern of your own: match a version string like v1.12.3 (hint: escape the dots, or they match anything).
- Keep the file open — the next two exercises extend it.
Groups — from matching to extracting
15 min- Extend the lab with the starter code below: a log-line pattern with three NAMED groups. Run it and inspect the dict each line produces.
- Walk the pattern left to right and annotate every piece with a comment: what does (?P<date>\d{4}-\d{2}-\d{2}) capture? Why [A-Z]+ for the level? Why does message use .* and why is it safe HERE (anchored at the end, nothing after it)?
- The if m: guard matters — feed the pattern the junk line and confirm it returns None instead of crashing. Never call .group on a None: that AttributeError is the most common regex crash in production code.
- re.sub practice: redact both email addresses in the text to <redacted>, then a harder one: keep the domain, redacting only the user part (capture the domain in a group, use \g<domain> in the replacement). PII redaction, ten weeks early.
- Convert your findall calls from exercise 1 into finditer where you use the match objects — print each match's .start() position too. finditer is the streaming-friendly form you'll want tomorrow.
Greed, laziness & verbose patterns
15 min- New section in the lab: run the greedy-vs-lazy demo on the tags string. Three patterns, three results — greedy .*, lazy .*?, and the specific [^>]+. Write one line on why the specific version is the professional default (it can't backtrack pathologically and it documents intent).
- Feel the over-match in realistic data: extract quoted strings from the config line with "(.*)" (wrong — one giant match) then "([^"]*)" (right). Quoted-field extraction is where greed bites everyone once.
- Rewrite the log-line pattern from exercise 2 in re.VERBOSE form using the starter below — same pattern, now with comments and breathing room. This is how patterns get code-reviewed.
- When NOT to use regex, hands-on: try to imagine matching nested brackets like ((a(b))c) — write in a comment why no regex can count nesting depth (finite automata have no stack). Then note the right tools: json.loads for JSON, an HTML parser for HTML.
- Speed habit: for the pattern used in a loop over 200k lines tomorrow, confirm you compiled it once outside the loop.
On your own
The extraction gauntlet
20 minBuild gauntlet.py against the messy text below (paste it as a triple-quoted string):
"Order #4471 shipped to ada@example.com on 2026-08-11. Card ending 4242. Support: +1-555-0134 or +44 20 7946 0958. Order #4472 pending. Ref: v2.1.0-beta. Visit https://status.example.com/incidents?id=99 for updates. Contact grace.hopper@lab.example.org (backup: g.hopper@navy.mil)."
Goal: extract, each with its own compiled pattern: (1) all order numbers; (2) all email addresses — including dotted user parts; (3) all dates; (4) the URL; (5) all phone numbers (both formats — this one is genuinely hard; get the two shown, don't chase perfection); then (6) produce a PII-safe version: emails and card reference redacted, order numbers kept.
Constraints: raw strings everywhere; named groups where you extract sub-parts; every pattern preceded by a comment stating its shape in English.
Hints (only if stuck): phone numbers — alternation of two explicit shapes beats one clever mega-pattern. Perfect email regex doesn't exist (truly matching the spec takes pages); "good enough for this text" is the professional answer — write down that lesson.
logparse.py — the parser Day 14 will import
Build tomorrow's engine today: logparse.py with a compiled VERBOSE pattern for the app.log format from Day 6 (date, time, level, IP, message) using named groups; a function parse_line(line) returning the groupdict (with the time field split out) or None for junk; and parse_lines(lines) — a GENERATOR (Day 11) that takes an iterable of lines and yields parsed dicts, silently counting junk lines in a parse_lines.skipped attribute or returned counter. Include a main-guard demo running it over app.log via a streaming pipeline and printing the first 5 parsed dicts (islice) plus the junk count. Add asserts for three sample lines (one of each level) and one junk line — your first taste of Day 18's testing habit. Commit it.
Common mistakes & misconceptions
- Forgetting the r prefix. "\d" survives by luck but "\b" becomes a backspace character and your word-boundary pattern silently never matches. Raw strings for every pattern, no exceptions.
- Calling .group() on the result of a failed search. re.search returns None on no-match; guard with if m: first. The AttributeError-on-None is the signature regex crash.
- Using .* between fields and wondering why it swallowed half the line. Greedy quantifiers grab maximally; prefer the specific class ([^,]+, [^>]+) that states what the field CAN contain.
- findall with groups returning tuples (or just the group) when you expected whole matches. Groups change findall's return shape — use finditer with .group(0) when you want full matches AND groups.
- Parsing HTML/JSON/nested anything with regex. No regex can count nesting. json.loads and real parsers exist; regex handles flat, line-shaped text.
- Gold-plating patterns to handle every theoretical input (the "perfect email regex"). Match the data you actually have, guard the None case, count the misses — the junk counter tells you if reality disagrees.
Q1. Which strings does r"\d{4}-\d{2}" fully match?
Q2. On "<b>hi</b>", the pattern <.*> matches…
Q3. You need each match's captured fields while streaming a 5 GB log. Best verb?
Go deeper — curated resources
- Catastrophic backtracking — Patterns like (a+)+$ on a long non-matching string can run for centuries — ambiguous nesting makes the engine try exponential paths. This is a real DoS class (ReDoS). The fix is the habit you learned today: specific classes over stacked greedy quantifiers.
- All three labs run with predictions written before outputs
- Gauntlet extracts all six targets; the "good enough" lesson written down
- logparse.py asserts pass; streaming demo works on app.log
- Redaction exercise produces the PII-safe text with domains kept
- Quiz ≥ 2/3 (redo the greed demo if you missed question 2)
← Back: Regex picks up where Day 5's string methods stopped — split and startswith for structure you know, patterns for structure you must DESCRIBE. parse_lines is a Day 11 conveyor belt with a pattern inside, and the raw log is Day 6's own app.log.
Forward →: Day 14's analyzer imports logparse.py unchanged. Redaction patterns return as Day 132's PII guardrails; log forensics with patterns is Day 172's customer-debugging bread and butter; and when LLMs emit almost-structured text, Day 110 pairs schema validation with exactly these extraction fallbacks.
Unlocks: D14 Week 2 Checkpoint: Log Analyzer