Day 68 · Mise en place

Cleaning & Validation

You will be able to

Choose a defensible missing-data strategy (drop, impute, flag+impute) per column, per mechanism
Detect and repair sentinels, unit mix-ups, duplicates, and impossible values
Encode a data contract as a pydantic model and collect violations as a report, not a crash
Write a reproducible raw→clean pipeline that logs every action and never mutates the raw data
Argue the garbage-in economics of cleaning to a stakeholder

Today's ~120 minutes

Spaced-rep warm-up: Day 67 cards (EDA stages, leakage)	10 min
ELI5 + tech read: mechanisms, repair bench, contracts	20 min
Guided: diagnose the dirt + contract-then-clean	42 min
Practice: the unit trap, solo	18 min
Project: clean.py + cleaning report	20 min
Quiz + flashcards	10 min

Builds on: Day 67 — EDA — finding the dirt · Day 41 — pydantic models & validation · Day 66 — Wrangling pipelines

The analogy

🧸 ELI5⚙ Tech

Watch a professional kitchen before service: nothing is cooking, yet everyone is working. Washing, chopping, labeling, lining up ingredients in bins — mise en place, "everything in its place". Amateurs skip it and improvise mid-cook; that is how garlic burns while you frantically dice an onion. Professionals know the boring preparation IS the cooking.

Data cleaning is mise en place, and it has two halves. The prep itself: fill or discard the gaps, fix the mislabeled jars (a column where half the values are in cents and half in dollars), throw out the duplicate deliveries, decide what to do with the weird ones (is a 250-year-old customer a typo, or a test account?). And then the allergy check at the pass: a written contract — every plate that leaves this kitchen has been checked against the ticket. In code, that contract is a schema: age is an integer between 0 and 120, plan is one of three known values, signup_date is a real date in the past. Every row is checked; violations get collected into a report instead of poisoning dinner. The deepest rule of the kitchen: never chop on the original — the raw file stays untouched, and the cleaning is a script you can rerun when next month's delivery arrives just as dirty.

Why this matters on the job

"Garbage in, garbage out" has a price tag: a model trained on a column with mixed cents-and-dollars learns nonsense, ships, and misprices something real — and the postmortem finds the bug was visible in row 3 of the CSV. Cleaning-as-a-script matters doubly for FDEs: customer data arrives dirty EVERY month, and a pipeline that validates loudly turns "your integration broke our dashboard" into "our contract caught 212 bad rows from your export — here is the report". The capstone's document-ingest (Day 119) and the eval-set hygiene of Day 134 are this discipline aimed at text.

Guided practice

guided 1

Diagnose the dirt, systematically

20 min

Paste the starter — it generates a clean seeded dataset and then corrupts it the way real systems do: sentinel -999s and "unknown"s, 18 duplicated rows, a cents/dollars unit mix from "source B", and a handful of impossible ages.
Run the audit in order, recording counts for each: (a) value_counts spikes → find the -999s in age and "unknown"s in region; (b) df.describe() → the min age of -999 and max age of 240 jump out; (c) df.duplicated().sum() → exact dupes; (d) histogram of amount → two humps, ~100× apart. Group by source and compare medians to confirm the unit hypothesis (B is in cents).
For each of the four problems, write ONE line: symptom → root cause → chosen repair. This table is the core of a cleaning report.
Convert sentinels to real NaN (df.age.replace(-999, np.nan), df.region.replace("unknown", np.nan)) and re-run isna().mean() — the true missingness rate was hiding behind the sentinels.
Do NOT fix anything else yet — diagnosis and repair are separate stages, and the repair belongs in the scripted pipeline of exercise 2.

🐍 python — editable, runs in your browser

import numpy as np
import pandas as pd

rng = np.random.default_rng(seed=68)
N = 800
df = pd.DataFrame({
    "customer_id": np.arange(1, N + 1),
    "age": rng.integers(18, 80, size=N).astype(float),
    "region": rng.choice(["north", "south", "east", "west"], size=N),
    "source": rng.choice(["A", "B"], size=N, p=[0.6, 0.4]),
    "amount": np.round(rng.lognormal(3.5, 0.6, size=N), 2),
    "signup": pd.Timestamp("2023-01-01")
              + pd.to_timedelta(rng.integers(0, 700, size=N), unit="D"),
})

# corruption, as real systems do it
df.loc[rng.random(N) < 0.06, "age"] = -999            # sentinel
df.loc[rng.random(N) < 0.04, "region"] = "unknown"    # sentinel
df.loc[df.source == "B", "amount"] *= 100             # cents vs dollars
df.loc[rng.choice(N, 5, replace=False), "age"] = rng.integers(150, 250, 5)  # impossible
df = pd.concat([df, df.sample(18, random_state=1)], ignore_index=True)      # dupes

print(df["age"].value_counts().head(3))       # the -999 spike
print(df.describe()[["age", "amount"]])
print("exact duplicates:", df.duplicated().sum())
print(df.groupby("source")["amount"].median()) # ~100x apart -> unit bug

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Contract first, then clean

22 min

Write the contract as a pydantic model (starter below): types, ranges, an enum for region, and amounts in dollars with a sane ceiling.
Validate every raw row and COLLECT the failures: a loop that appends (row index, error summary) instead of raising. Print the violations report grouped by field — age and amount should dominate, matching your diagnosis.
Now build clean(df) as a pure function applying the repairs in a fixed order: sentinels→NaN, dedupe (log count), unit-fix source B (divide by 100), null out impossible ages, median-impute age WITH an age_was_missing flag, mode-impute region. Log rows and repair counts at every step.
Re-validate the CLEANED frame against the contract: violations should drop to zero. That before/after pair of numbers — 212 violations → 0 — is the headline of a cleaning report.
Assert the endgame: no NaNs in contracted columns, amounts unimodal (p99/median under ~30), no duplicate customer_ids. The asserts are the contract's enforcement arm inside the pipeline.
Rerun the whole script and confirm identical output — the raw frame was never mutated (clean() returned a copy), so the pipeline is rerunnable forever.

🐍 python — editable, runs in your browser

import numpy as np
import pandas as pd
from pydantic import BaseModel, Field, ValidationError
from typing import Literal

class CustomerRow(BaseModel):
    customer_id: int = Field(gt=0)
    age: float = Field(ge=18, le=110)
    region: Literal["north", "south", "east", "west"]
    amount: float = Field(gt=0, lt=2000)   # dollars, sane ceiling

def violations(frame):
    report = []
    for i, row in enumerate(frame[["customer_id", "age", "region", "amount"]]
                            .to_dict("records")):
        try:
            CustomerRow(**row)
        except ValidationError as e:
            for err in e.errors():
                report.append((i, err["loc"][0], str(row[err["loc"][0]])[:12]))
    return pd.DataFrame(report, columns=["row", "field", "value"])

bad = violations(df)
print("violations by field:\n", bad.field.value_counts())

def clean(raw):
    out = raw.copy()
    out["age"] = out["age"].replace(-999, np.nan)
    out["region"] = out["region"].replace("unknown", np.nan)
    n0 = len(out); out = out.drop_duplicates(subset="customer_id", keep="first")
    print(f"dedupe: dropped {n0 - len(out)}")
    out.loc[out.source == "B", "amount"] /= 100
    out.loc[out.age > 110, "age"] = np.nan
    out["age_was_missing"] = out["age"].isna()
    out["age"] = out["age"].fillna(out["age"].median())
    out["region"] = out["region"].fillna(out["region"].mode()[0])
    return out

cleaned = clean(df)
print("violations after clean:", len(violations(cleaned)))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

The unit trap, solo

18 min

You receive a payments extract (build it seeded: 600 rows, columns ["payment_id", "provider", "amount"], providers "stripe" and "legacy") where ONE provider reports in cents — but this time nobody told you which, or whether it is true at all.

Your goals: (1) detect the problem from the data alone — a histogram of log10(amount) shows two humps; quantify the separation (median by provider, ratio ≈ 100); (2) state the evidence for which provider is in cents in two sentences (magnitude ratio + which range is plausible for real payments); (3) fix it, then PROVE the fix: the combined log-histogram is unimodal and the provider medians agree within noise; (4) add the regression guard — an assert on the median ratio being within [0.5, 2] — that would catch this bug in next month's file automatically.

Hints: np.log10 makes multiplicative gaps additive (Day 58's log-normal lesson); the proof step matters as much as the fix — "I divided by 100" is not evidence the data is now right. This exact bug, in production, is a mispriced invoice.

Ship before you stop

clean.py + the cleaning report

Assemble today into the deliverable shape: clean.py with three parts — generate_raw() (the seeded dirty dataset), validate(df) returning a violations DataFrame, and clean(raw) as a pure, logged function — plus cleaning_report.md: the four-row symptom→cause→repair table from guided 1, the before/after violation counts, row counts at each stage, and a three-sentence "garbage-in economics" paragraph for a stakeholder (what the unit bug would have cost downstream, why the contract now catches it monthly, what the supplier should fix at the source). Commit both — Day 70's checkpoint requires exactly this cleaning evidence, and Day 83's churn project reuses clean.py's structure verbatim.

Rubric — check what you completed (0/6)

clean() is pure (never mutates raw), logged, and rerunnable with identical outputValidation report shows nonzero violations on raw and zero on cleanedMissing-data choices are per-column and justified in the report (median vs mode vs flag)The was_missing indicator column exists and is documented as a feature candidateReport includes the symptom→cause→repair table and the economics paragraphCommitted to the practice repo

Common mistakes & misconceptions

Imputing before investigating WHY values are missing. Missingness concentrated in a segment (or caused by the value itself) biases naive imputation — mechanism first, method second.
Dropping every row with any NaN. On wide tables this can discard most of the data and biases toward complete-record customers — the opposite of random.
Mean-imputing skewed columns. The mean is tail-dragged (Day 58); the median is the defensible default for numerics.
Treating sentinels as values: averaging ages with -999s in them, or counting "unknown" as a region. Convert to NaN first, then decide.
Deduplicating on the full row when the KEY is what must be unique — two rows with the same ticket_id but different timestamps need a keep-policy, not drop_duplicates() defaults.
Cleaning in an interactive session and overwriting the raw file. If the raw data is gone, so is your ability to fix your own mistakes — raw is read-only, cleaning is a script.

Knowledge check

Q1. csat is missing mostly for basic-tier customers (as found yesterday). The best-practice first move is…

Q2. A histogram of a money column shows two clean humps about 100× apart, split perfectly by source system. Diagnosis?

Q3. Why validate rows with a pydantic contract that COLLECTS errors instead of raising on the first one?

Go deeper — curated resources

docspandas User Guide — working with missing data ↗25 min courseKaggle Learn — Data Cleaning (interactive) ↗30 min courseGoogle ML Crash Course — data preparation modules ↗20 min

If you have a third hour

Great Expectations & pandera — data contracts at scale — Today's pydantic pattern industrialized: declarative expectation suites, data docs, and CI integration for datasets. Skim the concepts — the capstone's ingest checks (Day 119) borrow this shape.

Done means

All four planted problems diagnosed with the symptom→cause→repair table
Pydantic contract written; violations report shows raw count → 0 after cleaning
Unit trap solved solo with detection evidence, proof of fix, and a regression assert
clean.py + cleaning_report.md committed; quiz ≥ 2/3

How this connects

← Back: You are repairing exactly what Day 67's interview uncovered; the pipeline shape (pure stages, logged counts) is Day 66's wrangle.py; the pydantic contract is Day 41's request validation aimed at data files; median-over-mean is Day 58.

Forward →: Day 69 builds features on the cleaned output (the was_missing flag becomes a feature; imputation moves INSIDE the pipeline to avoid leakage). Day 70's report requires the cleaning evidence, and Day 119's capstone ingest validates documents with this same contract pattern.

Unlocks: D69 Feature Engineering · D70 Week 10 Checkpoint: EDA Report · D110 Structured Outputs — Forms, Not Essays