Day 69 · Cutting ingredients so the pan can cook them

Feature Engineering

You will be able to

Encode categoricals appropriately: one-hot vs ordinal vs target encoding, with the trade-offs
Scale numeric features and state which model families care and which do not
Extract predictive features from datetimes (including cyclical encodings) and raw text columns
Explain train/test fitting discipline: transformers learn on train only
Build a leak-proof preprocessing pipeline with sklearn ColumnTransformer

Today's ~120 minutes

Spaced-rep warm-up: Day 68 cards (mechanisms, contracts)	10 min
ELI5 + tech read: encodings, scaling, the fitting discipline	20 min
Guided: encode/scale by hand + ColumnTransformer	42 min
Practice: the leak hunt	18 min
Project: features.py feature factory	20 min
Quiz + flashcards	10 min

Builds on: Day 68 — Cleaned, validated data · Day 64 — NumPy vectorized transforms · Day 67 — EDA — which signals are real

The analogy

🧸 ELI5⚙ Tech

A pan can only cook what fits in it. Nobody sears a whole pumpkin; you cut it into pieces sized for the heat to reach the middle. Models are the pan: they eat fixed-size rows of numbers, and they can only "reach" patterns that the cut exposes. A raw timestamp is a whole pumpkin — one giant number the model can barely bite. Cut it into hour-of-day, day-of-week, is-weekend, and suddenly the pattern ("support tickets spike Monday mornings") is bite-sized. Feature engineering is knife work: same ingredients, cut so the learning can cook.

Different dishes need different cuts. Category labels like "pro/basic/enterprise" become separate yes/no columns (one-hot) — unless the categories have a true order. Numbers on wildly different scales (tenure 1–60, spend 20–4,000) get standardized so distance-based learners don't think spend is 60× more important. And one knife rule is absolute, the same one from yesterday's kitchen: you season from the TRAIN shelf only. If your scaler learns the mean of the whole dataset — test rows included — your model has tasted the exam. The fix is not vigilance, it is machinery: a pipeline object that physically cannot fit on data you didn't hand it.

Why this matters on the job

On tabular data, better features beat fancier models — the winning move in Day 77's mini-competition and Day 83's churn project is almost always a feature, not a hyperparameter. The leakage discipline is career-critical: "fit the scaler on everything" is the most common silent bug in applied ML, inflating offline metrics and detonating in production. And the concept generalizes: prompt context selection (Day 108) and chunking for RAG (Day 114) are feature engineering for LLMs — deciding what the model gets to see, cut to the size it can digest.

Guided practice

guided 1

Encode and scale by hand — and watch the leak happen

20 min

Paste the starter — the cleaned churn-style dataset from this week, seeded, split 80/20 into train/test FIRST (before any transformer touches it).
One-hot the plan column with OneHotEncoder(handle_unknown="ignore"): fit on train, transform both. Print the feature names. Then simulate the future: transform a row with plan="platinum" (never seen) and confirm it becomes all-zeros instead of an exception.
Ordinal-encode plan as basic=0, pro=1, enterprise=2 — defensible, there is a real order. Now ordinal-encode region alphabetically and write one sentence on what a linear model would wrongly conclude ("west is 3× more region than east").
Scale monthly_spend with StandardScaler fit on TRAIN; check test-transformed mean is NOT exactly 0 (it shouldn't be — test is scaled by train's parameters).
Now commit the classic crime deliberately: fit the scaler on the FULL dataset, and compare the transformed train values against the honest version. The difference is small — which is exactly why this bug survives review. Write the rule: the size of the leak is not the point; the direction of information flow is.

🐍 python — editable, runs in your browser

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(seed=69)
N = 1000
df = pd.DataFrame({
    "plan": rng.choice(["basic", "pro", "enterprise"], size=N, p=[0.55, 0.33, 0.12]),
    "region": rng.choice(["north", "south", "east", "west"], size=N),
    "tenure_months": rng.integers(1, 60, size=N),
    "monthly_spend": np.round(rng.lognormal(3.4, 0.8, size=N), 2),
    "signup_hour": rng.integers(0, 24, size=N),
})
df["churned"] = rng.random(N) < np.clip(
    0.45 - 0.005 * df.tenure_months + 0.1 * (df.plan == "basic"), 0.05, 0.9)

train, test = train_test_split(df, test_size=0.2, random_state=69)

ohe = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
ohe.fit(train[["plan"]])
print(ohe.get_feature_names_out())
unseen = pd.DataFrame({"plan": ["platinum"]})
print("unseen category ->", ohe.transform(unseen))   # all zeros, no crash

scaler = StandardScaler().fit(train[["monthly_spend"]])
test_scaled = scaler.transform(test[["monthly_spend"]])
print("test mean after train-fit scaling:", test_scaled.mean().round(3))

leaky = StandardScaler().fit(df[["monthly_spend"]])   # the crime
print("honest vs leaky train mean:",
      scaler.mean_[0].round(2), "vs", leaky.mean_[0].round(2))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

ColumnTransformer — the whole kitchen in one object

22 min

Add the datetime knife work as plain pandas first (features are created before the transformer; the transformer handles impute/scale/encode): hour_sin = sin(2π·signup_hour/24), hour_cos = cos(...). Verify the payoff: compute the distance between hour 23 and hour 1 in raw form (22 apart) vs sin/cos form (close) — the encoding restored the clock's geometry.
Add two more cut-by-hand features: spend_per_tenure = monthly_spend / tenure_months (an interaction) and tenure_bin via pd.cut (0–12 / 13–24 / 25+).
Build the ColumnTransformer: numeric columns → SimpleImputer(median) + StandardScaler in a Pipeline; categorical columns (plan, region, tenure_bin) → SimpleImputer(most_frequent) + OneHotEncoder(handle_unknown="ignore"). Passthrough hour_sin/hour_cos (already in range).
fit_transform on train, transform on test. Print the output shape and get_feature_names_out() — read the list aloud; this is EXACTLY what the model will eat, and being able to name every column is a debugging superpower.
Prove the leak-proofing: the transformer, once fit, contains train-derived parameters only — print named_transformers_ internals (the scaler's mean_, the encoder's categories_). One object, one fit, zero opportunities to season from the test shelf.
Note what did NOT go in: refund_issued (Day 67's leak) is excluded by design, and the exclusion is documented in the feature list.

🐍 python — editable, runs in your browser

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

for frame in (train, test):
    frame["hour_sin"] = np.sin(2 * np.pi * frame.signup_hour / 24)
    frame["hour_cos"] = np.cos(2 * np.pi * frame.signup_hour / 24)
    frame["spend_per_tenure"] = frame.monthly_spend / frame.tenure_months
    frame["tenure_bin"] = pd.cut(frame.tenure_months, bins=[0, 12, 24, 60],
                                 labels=["0-12", "13-24", "25+"])

num_cols = ["tenure_months", "monthly_spend", "spend_per_tenure"]
cat_cols = ["plan", "region", "tenure_bin"]
cyc_cols = ["hour_sin", "hour_cos"]

pre = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc", StandardScaler())]), num_cols),
    ("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                      ("oh", OneHotEncoder(handle_unknown="ignore"))]), cat_cols),
    ("cyc", "passthrough", cyc_cols),
])

X_train = pre.fit_transform(train)
X_test = pre.transform(test)          # transform ONLY — no fitting on test
print("train:", X_train.shape, " test:", X_test.shape)
print(list(pre.get_feature_names_out()))
print("scaler means (train-derived):",
      pre.named_transformers_["num"].named_steps["sc"].mean_.round(2))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

The leak hunt

18 min

Three preprocessing setups cross your code review. For each: leak or clean? Name the information flow, rank severity, and write the fix.

A. scaler.fit(X) on the full dataset, then train/test split, then a model trained on the scaled train half. B. Target encoding: plan_churn_rate = df.groupby("plan")["churned"].mean() computed on ALL rows, mapped onto both train and test as a feature. C. imputer.fit(X_train); X_test = imputer.transform(X_test); model evaluated on X_test.

Then the transfer question: your golden-set eval (Day 134 foreshadow) uses few-shot examples in the prompt. What is the analogous crime, and what is the analogous rule? (Answer shape: examples drawn from the eval set itself = fitting on test; the rule: eval cases must never appear in the prompt.)

Hints: B is the worst — the feature literally contains averaged test-set TARGETS, not just statistics of inputs; A is real but mild (input statistics only); C is clean and is the exact pattern the ColumnTransformer mechanizes. Severity follows how much target information crosses the line.

Ship before you stop

features.py — the feature factory

Build features.py in your practice repo, structured for reuse in Days 70, 71, and 83: add_features(df) (pure function: cyclical hour, spend_per_tenure, tenure_bin, text-length features if a text column exists, the Day 68 was_missing flags kept), build_preprocessor(num_cols, cat_cols, passthrough) returning the ColumnTransformer, and a FEATURES.md block (docstring or file) listing every output feature with one line each: source column, transform, and why it should carry signal — plus an explicit EXCLUDED section naming refund_issued and the reason. Demo at the bottom: split, fit on train, transform both, print shapes and feature names. Commit it.

Rubric — check what you completed (0/6)

add_features is pure and covered by at least three asserts (shapes, ranges, no NaN introduced)Preprocessor fits on train only in the demo; test is transform-onlyCyclical encoding verified (hour 23 and hour 1 are near neighbors in sin/cos space)Feature documentation lists every column with a one-line rationaleEXCLUDED section names the leak and whyCommitted to the practice repo

Common mistakes & misconceptions

Fitting any transformer (scaler, imputer, encoder) on the full dataset before splitting. Information flows from test into training — offline metrics inflate, production deflates. Split first, fit on train, always.
Ordinal-encoding unordered categories. The model reads the integers as magnitudes — "west > east" — and linear/distance models act on the fiction. One-hot unless a true order exists.
One-hot encoding a 10,000-value ID column into 10,000 features. High cardinality wants target encoding (done out-of-fold), frequency encoding, or exclusion.
Scaling everything for a tree model and believing it mattered. Trees split on thresholds; scaling is a no-op for them — know WHICH families care (distance- and gradient-based).
Naive target encoding computed on all rows — averaged test TARGETS become a training feature; the most concentrated leak in tabular ML.
Dropping the datetime column instead of decomposing it. The signal ("weekend signups churn more") lives in the parts, not the raw timestamp.

Knowledge check

Q1. Why must StandardScaler be fit on the training split only?

Q2. Encoding hour-of-day as sin(2πh/24) and cos(2πh/24) exists to…

Q3. Which model family is essentially indifferent to feature scaling?

Go deeper — curated resources

docsscikit-learn — preprocessing data (official user guide) ↗30 min docsscikit-learn User Guide — pipelines & ColumnTransformer ↗20 min courseKaggle Learn — Feature Engineering (interactive) ↗30 min courseGoogle ML Crash Course — feature engineering modules ↗20 min

If you have a third hour

Out-of-fold target encoding — The safe version of the powerful trick: encode each row using target means computed WITHOUT that row's fold. Day 77's competition is where you may want it; read the idea now so it is not magic then.

Done means

Unseen-category behavior and the ordinal-encoding fiction both demonstrated
ColumnTransformer built; feature names printed and every column explainable
Leak hunt: all three setups classified correctly with severity ranking
features.py committed with documentation and the EXCLUDED section; quiz ≥ 2/3

How this connects

← Back: The features stand on Day 68's cleaned data (the was_missing flag survives as signal); log1p-before-scaling is Day 58's tails; vectorized sin/cos columns are Day 64's broadcasting; and the excluded refund_issued is Day 67's leak, formally banished.

Forward →: Day 71 feeds this exact matrix into sklearn models — your first fit/predict runs on today's output. Day 77's competition is won with feature work, Day 82 wraps preprocessor+model into one deployable pipeline, and Day 92 reframes features as learned embeddings.

Unlocks: D70 Week 10 Checkpoint: EDA Report · D71 ML Framing & Linear Regression · D72 Logistic Regression & Losses · D78 Clustering