Day 83 · The whole assembly line

Phase Project: Churn Prediction End-to-End

You will be able to

Frame a business problem as an ML problem: target, metric, and baseline chosen from the harm model
Execute the full pipeline solo: EDA → features → bake-off → error analysis → shipped artifact
Produce a stakeholder-readable report where every claim carries a number and an error bar
Self-grade against an 8-criterion rubric and log honest gaps

Today's ~120 minutes

Milestone 1: setup, EDA, hazard hunt, metric declaration	35 min
Milestone 2: features, tracked bake-off, selection	40 min
Milestone 3: error analysis, seal-break, report, self-grade	35 min
Practice: the skeptical-VP drill	10 min

Builds on: Day 82 — The ml-template & inference contract · Day 81 — Tracked runs · Day 80 — Error-analysis playbook · Day 77 — Competition protocol & model card

The analogy

🧸 ELI5⚙ Tech

Every station on the assembly line, you have run alone: framing (Day 71), features (Day 69), models (Days 71–74), metrics (Day 75), validation (Day 76), error analysis (Day 80), tracking (Day 81), structure (Day 82). Today the conveyor belt runs the whole line at once, and you are every worker on it — because that is what the job actually is. Nobody ships "a decision tree"; they ship an end-to-end answer to a business question, with the receipts.

The scenario: a subscription company bleeds ~19% of customers a year and wants to know who will leave BEFORE they leave, so the retention team can intervene. Your deliverable is not a model — it is a decision-support system: a prediction pipeline the team could call, a report their manager could read, and an honest account of where it fails. The twist that makes this a rite of passage: no day-number handrails. The brief says what is needed; YOU decide which tool from the last twelve days answers each part. That transfer — from "following the lesson" to "selecting from your toolkit" — is the entire point of a phase project.

Why this matters on the job

Churn prediction is the single most common commercial ML brief — telecom, SaaS, banking, streaming all buy it — which is exactly why it is this phase's graded project and a stock interview case study ("how would you build a churn model?" is asked verbatim). Everything here is portfolio-grade: a repo with tracked experiments, a tested pipeline, and a model card demonstrates the difference between "took a course" and "does the work." And the shape of today — brief in, scoped system + honest report out, alone, in hours — is a deliberate rehearsal of an FDE engagement and of Day 174's full simulation.

Guided practice

guided 1

Milestone 1 — Setup, data, EDA, metric declaration (target: 35 min)

35 min

Copy ml_template/ to churn_project/; follow your own porting checklist from the Day 82 README. Save the starter generator as src/churn/data.py's loader.
Generate the data; seal the test split immediately in train.py (random_state=83).
Timeboxed EDA (30 min hard stop, Day 67 checklist): churn rate, distributions, churn rate BY each feature, correlations. You are hunting the two planted hazards: a column that would let the model memorize individuals, and a column that could not be known at prediction time. Decide the fate of each and write the decision + reasoning into REPORT.md's data-findings section.
Declare the metric in REPORT.md now, with the harm-model sentence: what does a missed churner cost vs a wasted retention offer? Choose headline + operating point accordingly.

🐍 python — editable, runs in your browser

# src/churn/data.py — synthetic churn generator (standalone, offline)
import numpy as np
import pandas as pd

def load_data(cfg=None, n=8000, seed=83):
    rng = np.random.default_rng(seed)
    contract = rng.choice(["month-to-month", "one-year", "two-year"],
                          n, p=[0.55, 0.25, 0.20])
    tenure = rng.integers(1, 72, n).astype(float)
    monthly = rng.normal(65, 22, n).clip(15, 160)
    tickets = rng.poisson(1.2, n) + (rng.random(n) < 0.15) * rng.poisson(4, n)
    payment = rng.choice(["card", "bank", "check"], n, p=[0.5, 0.3, 0.2])
    usage_trend = rng.normal(0, 1, n)          # + = rising usage

logit = (-1.9
             + 1.1 * (contract == "month-to-month")
             - 0.35 * (contract == "two-year")
             - 0.030 * tenure
             + 0.012 * (monthly - 65)
             + 0.22 * tickets
             - 0.45 * usage_trend
             + 0.35 * (payment == "check"))
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

# hazard 1: an ID column (memorization bait)
    customer_id = np.arange(100000, 100000 + n)
    # hazard 2: leakage — computed AFTER the churn window
    last_login_days = np.where(
        churn == 1, rng.integers(30, 90, n), rng.integers(0, 21, n))

return pd.DataFrame({
        "customer_id": customer_id, "contract": contract, "tenure": tenure,
        "monthly_charges": monthly.round(2), "support_tickets": tickets,
        "payment_method": payment, "usage_trend": usage_trend.round(3),
        "last_login_days": last_login_days, "churn": churn})

if __name__ == "__main__":
    df = load_data()
    print(df.shape, " churn rate:", round(df["churn"].mean(), 3))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Milestone 2 — Features, bake-off, selection (target: 40 min)

40 min

Build the pipeline in features.py: numeric passthrough/scale as your lineup needs, categoricals encoded per Day 77's rule of thumb. The hazard columns get whatever fate you decided — the config's feature lists make exclusion a one-line, documented act.
Run the tracked bake-off on the training portion: dummy, scaled logistic, random forest, HistGradientBoosting — CV mean ± std for your declared metric (scoring="average_precision" if you chose PR-AUC), every run to MLflow.
One tuning pass on the leader only (fair-fight rule from Day 74 — note it in the report). Use your toolkit's tune().
Select the ship candidate on CV evidence. If the boosted model and forest are within each other's error bars, say so and choose on secondary grounds (calibration need, speed) — write the sentence.
Curiosity checkpoint (do NOT ship it): add customer_id and last_login_days back in a scratch run. Watch the score "improve." Write one line in the report's limitations: what that mirage would have done in production.

guided 3

Milestone 3 — Error analysis, seal-break, report, self-grade (target: 35 min)

35 min

Run the Day 80 playbook on out-of-fold or validation predictions: slice_report by contract and tenure-band, read 20 errors with confident_errors, count hypotheses.
Apply ONE fix the ledger justifies (a feature like tenure-per-dollar, a binning, more weight on a slice — your call), remeasure CV, and record the before/after honestly — including if it did not help.
Seal-break: score dummy, logistic, and your final model on the untouched test set, once. Build the leaderboard table.
Finish REPORT.md (all sections from the tech spec). Run predict.py's contract check: one high-risk and one low-risk fictional customer through predict_one — paste both into the report as the "what the retention team sees" demo.
Self-grade against all 8 rubric criteria, one sentence of evidence each. Log gaps as GitHub issues (or a TODO section) — unfixed-but-known beats silently missing. Commit; tag churn-v1.

On your own

The skeptical-VP drill

10 min

A skeptical VP reads your report and fires three questions. Write a 2–3 sentence answer to each, with numbers from YOUR run: (1) "Why should I believe this beats just calling everyone whose contract is month-to-month?" (2) "It flags 300 customers a month — how many are false alarms, and what does that cost me?" (3) "What would make this model quietly wrong six months from now?"

Hints: (1) is a demand for a rule-based baseline comparison — if you didn't run one, run it now (it is one line: predict churn = month-to-month). (2) reads precision at your operating threshold. (3) is drift: tenure distributions shift, retention offers change behavior, the leakage column's temptation returns.

Today's build

The deliverable & the 8-criterion rubric

Ship churn_project/ as a self-contained repo directory: config-driven training on the synthetic churn data (or Telco), one persisted pipeline honoring the inference contract, MLflow-tracked experiments, tests passing, and REPORT.md written for a retention-team manager. The rubric below is the grade — 2 points each, 16 total: 13+ = strong pass, 10–12 = pass with gaps logged, <10 = schedule a half-day revisit before Day 85. Self-grade honestly; Day 84's writeup asks you to defend two of your own scores.

Rubric — check what you completed (0/8)

Framing: metric + operating threshold declared BEFORE modeling, justified from the stated harm model in ≤ 3 sentencesHazards: both planted traps (ID memorization bait, post-outcome leakage column) found, correctly handled, and documentedProtocol: test set sealed first and scored exactly once; all model selection on CV evidence with mean ± std reportedBake-off: ≥ 4 entries including dummy and scaled-logistic baselines; final model beats both on the sealed setError analysis: slices + 20-error ledger executed; one ledger-justified fix applied with before/after numbers (honest if null)Engineering: template structure, config-driven runs, passing tests, persisted pipeline + metadata, working predict_one demoTracking: every experiment in MLflow with params, metrics, data hash, git tag — the winning run fully reconstructibleReport: one page, stakeholder-readable, all sections present, every claim numbered, limitations include the drift risk

Common mistakes & misconceptions

Spending the first hour polishing EDA plots. The timebox exists because the model card needs findings, not gallery pieces — 30 minutes, findings as sentences, move.
Letting customer_id or last_login_days into the features "because the score went up." That score is the mirage; the rubric pays for CATCHING them, and production would pay for missing them.
Choosing the metric after the bake-off. Declaration-before-modeling is criterion 1; a metric chosen to flatter results fails it even if the number is good.
Tuning every model in the lineup for hours. One tuning pass on the leader, fair-fight noted — this is a one-day build, and the rubric rewards protocol, not leaderboard heroics.
Reporting feature importances as "why customers churn." Day 73's caveats apply doubly in a causal-sounding report; say "associated with risk in this model," not "causes."
Skipping the self-grade or grading generously. Day 84 asks you to defend two scores; inflated grades make that defense — and the habit — worthless.

Knowledge check

Q1. The last_login_days column boosts CV score dramatically. Why must it be dropped anyway?

Q2. Your boosted model's CV PR-AUC is 0.61 ± 0.03; the forest's is 0.60 ± 0.04. The honest report says…

Q3. The retention team can contact 150 customers/month. This constraint primarily changes…

Go deeper — curated resources

docsscikit-learn User Guide — 8.1 Pipelines (reference while building) ↗10 min docsscikit-learn User Guide — 3.4 Metrics (PR-AUC & scoring strings) ↗10 min docsMLflow Documentation — tracking reference ↗10 min courseKaggle Learn — feature engineering ideas for tabular churn ↗15 min

If you have a third hour

Survival analysis — churn's other framing — Classification asks "will they churn this window?"; survival models ask "how long until churn?" — handling censoring properly and yielding retention curves. Worth knowing the name (Cox proportional hazards) when a customer asks about time-to-event.

Done means

Both planted hazards found and documented with the handling decision
Sealed test scored once; final model beats both baselines on the declared metric
REPORT.md complete with leaderboard, error-analysis story, and limitations
Self-grade ≥ 10/16 with evidence sentences; gaps logged
Repo committed and tagged churn-v1

How this connects

← Back: This day is Weeks 10–12 compressed: Day 67's EDA timebox, Day 69's leakage radar, Day 74's fair bake-off, Day 75's harm-model metric, Day 76's error bars, Day 77's protocol, Day 80's playbook, Day 81's tracking, Day 82's template. The toolkit you grew daily was for this.

Forward →: Day 84 defends this work in interview form. Day 90's MNIST lab reruns the same arc in deep learning, and the capstone (Day 119) is this day again at production scale — brief, protocol, artifact, report. The skeptical-VP answers preview Day 171's stakeholder craft.

Unlocks: D84 Week 12 Checkpoint + Interview Drill II