ML Framing & Linear Regression
- Define machine learning as a function family + loss + optimizer, and identify all three in any model
- Classify a problem as regression or classification and name the target and features
- Fit, predict, and score a scikit-learn model using the estimator API and a proper train/test split
- Establish a dumb baseline before any model and justify why it comes first
- Recognize underfitting from the gap between model capacity and data shape
| Spaced-rep warm-up: due cards from Weeks 9–10 (gradients, features) | 10 min |
| ELI5 + tech read; revisit the gradient-descent visualizer with new eyes | 20 min |
| Guided: baseline first + read the model | 35 min |
| Practice: the line that could not bend | 20 min |
| Project: ml_toolkit.py report card function | 25 min |
| Quiz + write flashcards | 10 min |
Builds on: Day 56 — Linear regression by hand (NumPy + gradient descent) · Day 55 — Gradient descent lab · Day 69 — Feature engineering & train/test discipline
You track your friends' rents against their apartment sizes and plot the dots. With a ruler, you eyeball a straight line through the cloud — not touching every dot, but capturing the trend. Now a friend mentions a 700-square-foot place, and you slide your finger along the ruler: "about 1,900 a month." You just did machine learning: you chose a shape (a straight line), a definition of "best" (close to the dots), and a way to find it (nudging the ruler until the misses look small).
That is the whole game, formalized. The line is the model, the total miss is the loss, and the nudging is the optimizer. On Day 56 you WERE the optimizer — you wrote the gradient-descent nudging yourself in NumPy. Today scikit-learn does the nudging in one line, and your job shifts to the part machines can't do: framing the question, choosing what counts as a miss, and checking the line against dots it has never seen.
Every ML system you will ship — churn models, rerankers, LLM-judge calibrators — is function + loss + optimizer underneath, and interviewers probe exactly this framing ("what's your loss? what's your baseline?"). The baseline habit is a career-saver: FDEs regularly discover mid-engagement that a customer's "AI opportunity" is beaten by predicting the historical average. Finding that out in hour one with a DummyRegressor, instead of week three with a tuned model, is the difference between credibility and embarrassment. On Day 83 your churn project is graded baseline-first.
Rolling downhill in fog — loss vs. weight, one step at a time
step 1 / 5The loss landscape: every possible weight value has a loss. We can't see the whole curve — only the slope where we stand.
Guided practice
Baseline first, model second
20 min- Create
ml_framing_lab.pyand paste the starter code. It uses the built-in diabetes dataset (442 patients, 10 features, target = disease progression) so it runs standalone. If your in-browser interpreter lacks scikit-learn, run locally:pip install scikit-learn. - Run it. Record four numbers: baseline MAE, baseline R², model MAE, model R².
- Sanity-check: baseline R² should be ≈ 0 (slightly negative on test is normal). Explain in one sentence why the mean predictor defines zero.
- Compute the improvement: what fraction did MAE drop versus the baseline? That sentence — "the model cuts average error from 66 to 44, a 33% improvement over predicting the mean" — is how you will report every model from now on.
- Change
random_statein the split to 0 and re-run. The scores move. Note this — it is why Day 76 replaces one split with cross-validation.
Read the model like a scientist
15 min- After fitting, print
model.coef_andmodel.intercept_. Each coefficient answers: "holding everything else fixed, one unit more of this feature changes the prediction by how much?" - Zip the coefficients with
load_diabetes().feature_namesand sort by absolute value. Which three features drive the prediction hardest? - Predict for one patient: take
X_test[0], reshape withX_test[0:1], and call predict. Then verify by hand: intercept + dot product of coefficients and features (Day 50's dot product, earning rent). - Caveat to write down: these features are pre-standardized in this dataset, so magnitudes are comparable. On raw data, a coefficient's size depends on the feature's units — comparing them requires scaling first (Day 69).
On your own
The line that could not bend
20 minGenerate curved data: x = np.linspace(-3, 3, 200), y = x**2 + np.random.normal(0, 0.5, 200). Reshape x to a column with x.reshape(-1, 1).
Goal: (1) split, fit LinearRegression, record train AND test R² — both should be terrible, and similar to each other; (2) explain in one sentence why this is underfitting, not overfitting; (3) fix it without changing the model class: add x**2 as a second feature column (np.column_stack) and refit. R² should jump above 0.9.
Hints (only if stuck): underfitting = bad on both splits; the model is still linear in its INPUTS — you changed the inputs, which is exactly what feature engineering (Day 69) is for.
Your regression report card function
Create ml_toolkit.py in your practice repo — a module you will grow all phase and reuse in the Day 83 churn project. Today it gets one function: evaluate_regression(model, X_train, X_test, y_train, y_test) that fits the model, and returns a dict with train/test MAE, train/test R², and the same numbers for a DummyRegressor baseline fitted on the same split. Add a print_report(results) helper that renders a small comparison table and flags "WARNING: model barely beats baseline" when test-MAE improvement is under 10%. Demonstrate it on the diabetes dataset in a __main__ block, and commit.
Common mistakes & misconceptions
- Scoring on training data and calling it accuracy. Memorization looks like skill; only held-out scores count. Always report both, labeled.
- Skipping the baseline. R² already encodes "versus the mean," but MAE does not — a churn model with 85% accuracy sounds great until you learn 85% of customers never churn (Day 75's accuracy trap).
- Thinking R² = 0.5 means "half the predictions are right." It means the model explains half the variance the mean predictor leaves. It is not a percentage of correct answers.
- Treating a negative test R² as a bug. It is a verdict: the model is worse than predicting the average — often a leakage-free sign of a broken feature set or a shuffled target.
- Fixing underfitting with more data. More rows of curved data will not help a straight line; underfitting needs more capacity or better features, overfitting needs the opposite.
- Passing a 1-D x to fit. sklearn wants X as (n_samples, n_features) — reshape(-1, 1) for a single feature; the error message will haunt you until this is reflex.
Q1. Your model scores R² = 0.92 on training data and R² = 0.31 on the test split. The FIRST correct conclusion is…
Q2. Why fit a DummyRegressor before any real model?
Q3. A straight line fit to clearly quadratic data gives poor scores on BOTH train and test. The standard fix is…
Go deeper — curated resources
- Closed-form vs gradient descent for linear regression — The normal equation solves OLS exactly in O(n·d²) — sklearn uses an SVD-based solver (Day 52's SVD at work). Gradient descent wins when d is huge or the loss has no closed form — which is every neural net (Day 86).
- Baseline and model metrics recorded from guided exercise, with the one-sentence improvement report written
- Underfitting demo run: both fixes understood, R² > 0.9 after adding the squared feature
- ml_toolkit.py committed with working evaluate_regression + print_report
- Quiz ≥ 2/3
← Back: On Day 56 you built exactly this model with raw NumPy and hand-written gradient descent — today sklearn compressed your week of math into fit(). Day 69's leakage discipline is why the split happens before anything touches the data.
Forward →: Tomorrow logistic regression reuses this API for classification. Day 76 replaces the single split with cross-validation, and Day 83's churn project is graded on the baseline-first protocol you built today. On Day 135, "baseline first" returns as the golden rule of LLM evals.
Unlocks: D72 Logistic Regression & Losses · D73 Decision Trees · D74 Ensembles — Forests & Boosting · D75 Evaluation Metrics