Day 74 · Asking a crowd

Ensembles — Forests & Boosting

You will be able to

Explain how bagging + random feature subsets turn many overfit trees into one strong forest
Contrast bagging (parallel, variance-reducing) with boosting (sequential, bias-reducing)
Run a fair bake-off between logistic regression, random forest, and gradient boosting
Tune the few boosting knobs that matter — learning_rate, tree depth, n_estimators via early stopping
Say why ensembles dominate tabular problems and when the linear model still wins

Today's ~120 minutes

Spaced-rep warm-up: Day 73 cards (memorization, importances)	10 min
ELI5 + tech read	20 min
Guided: fair bake-off + learning-rate trade	38 min
Practice: bagging by hand	20 min
Project: bakeoff() + model selection memo	22 min
Quiz + flashcards	10 min

Builds on: Day 73 — Decision trees & overfitting · Day 60 — Sampling & the law of large numbers · Day 71 — Baselines & fair evaluation

The analogy

🧸 ELI5⚙ Tech

Ask one person to guess the weight of an ox and you get a wild number. Ask a fair full of people and average their guesses, and — famously — the average lands within a percent of the truth. Each guesser is noisy, but their errors point in different directions, and averaging cancels them. The crowd is only wise, though, if the guessers are INDEPENDENT: let them confer and they converge on one confident, shared mistake.

A random forest is a manufactured crowd of decision trees. To keep the trees from all making the same mistake, it forces disagreement two ways: each tree trains on a different bootstrap resample of the data, and at every split each tree may only consider a random subset of features. Hundreds of diverse, individually-overfit trees then vote, and the memorization noise averages away. Boosting is a different kind of teamwork — a relay, not a crowd: each small tree studies the errors of the team so far and specializes in fixing them, with a learning rate keeping any one runner from sprinting off course.

Why this matters on the job

Gradient-boosted trees are still the reigning champions of tabular data — the churn tables, claims data, and transaction logs that make up most enterprise ML. Knowing that "for tabular, start linear, then boost trees; save deep learning for text/images" is a hiring-signal opinion you can now defend. Interviews love the bagging-vs-boosting contrast. And for an FDE, the bake-off you run today — same split, same metric, honest table — is the exact artifact that convinces a customer you chose their model on evidence, not fashion. Day 77's competition and Day 83's churn project are won with today's tools.

Guided practice

guided 1

The bake-off, run fairly

22 min

Paste the starter. It builds a synthetic-but-hard tabular dataset (20 features, only 8 informative, some redundant — realistic noise) and compares four models on one fixed split. Standalone; run locally if the browser lacks scikit-learn.
Run it. Record the table. Typical shape: dummy ≪ logistic < single tree < forest ≤ boosting — but verify, don't assume.
Fairness audit — answer in writing: same split for all models? Same metric? Any model given tuning the others were denied? (Here: none — all defaults. A tuned-vs-default comparison would be rigged.)
Increase the forest's n_estimators from 100 to 300. Score barely moves — diminishing returns after "enough trees." Now DECREASE to 5 and watch it wobble: too small a crowd.
Print the forest's oob_score_ (set oob_score=True) and compare it to the test score — a free validation estimate from the leftover bootstrap rows.

🐍 python — editable, runs in your browser

from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(
    n_samples=6000, n_features=20, n_informative=8, n_redundant=4,
    flip_y=0.03, class_sep=0.8, random_state=7
)
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.25, random_state=7, stratify=y
)

models = [
    ("dummy (majority)", DummyClassifier(strategy="most_frequent")),
    ("logistic reg", make_pipeline(StandardScaler(),
                                   LogisticRegression(max_iter=1000))),
    ("single tree", DecisionTreeClassifier(random_state=0)),
    ("random forest", RandomForestClassifier(
        n_estimators=100, oob_score=True, random_state=0, n_jobs=-1)),
    ("hist gradient boost", HistGradientBoostingClassifier(random_state=0)),
]

for name, m in models:
    m.fit(X_tr, y_tr)
    print(f"{name:20s} train={m.score(X_tr, y_tr):.3f}  test={m.score(X_te, y_te):.3f}")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

The learning-rate / n_estimators trade + early stopping

16 min

With the same data, fit HistGradientBoostingClassifier(learning_rate=lr, max_iter=500, early_stopping=True, validation_fraction=0.15, random_state=0) for lr in [1.0, 0.3, 0.1, 0.03].
For each run print lr, clf.n_iter_ (rounds actually used before early stopping fired), and test score. Pattern to find: smaller lr → more rounds needed → often slightly better test score, until it just gets slow.
lr = 1.0 is the cautionary row: fast, and usually worse — each tree over-commits, like a relay runner sprinting the wrong way.
Write the rule of thumb in your notes: set lr smallish (0.05–0.1), max_iter generous, and let early stopping choose the rounds. You tuned the most important boosting knob with four runs.

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Build bagging by hand

20 min

Prove the crowd effect without RandomForestClassifier. Train 25 unconstrained DecisionTreeClassifiers, each on its own bootstrap resample of the bake-off training data (indices via rng.choice(n, n, replace=True)). Average their predict_proba outputs and threshold at 0.5 for the ensemble prediction.

Goal: (1) report mean single-tree test accuracy vs your hand-rolled ensemble's accuracy — expect a clear jump; (2) plot or tabulate ensemble accuracy as you include 1, 5, 10, 25 trees — watch it climb then flatten; (3) two sentences: which ingredient created the gain, and what does RandomForest add that you didn't implement?

Hints: keep a list of fitted trees; np.mean([t.predict_proba(X_te)[:, 1] for t in trees], axis=0). The missing ingredient is per-split feature subsetting (max_features) — decorrelation.

Ship before you stop

The model bake-off report

Add bakeoff(models, X_train, X_test, y_train, y_test) to ml_toolkit.py: takes a list of (name, estimator) pairs, fits each on the shared split, and returns a sorted results table (train score, test score, gap, fit seconds via time.perf_counter). Include a DEFAULT_LINEUP constant with the five models from today's guided work. Then write bakeoff_notes.md: your table for today's dataset plus a six-sentence "model selection memo" a customer could read — which model you'd ship, why, and what you'd try next. This function is the engine of Day 77's competition and Day 83's churn bake-off. Commit both.

Rubric — check what you completed (0/5)

bakeoff runs any (name, estimator) list on one shared split and returns sorted results including fit timeGap (train − test) column present — overfitting visible at a glanceDEFAULT_LINEUP includes baseline, linear, single tree, forest, boostingMemo names a shipping choice and justifies it with numbers from the table, not vibesBoth files committed to the practice repo

Common mistakes & misconceptions

Believing more trees overfit a random forest. Averaging more bootstrap trees only stabilizes the estimate; n_estimators trades compute for variance, not test accuracy for train accuracy. (Boosting rounds are a different story — those DO overfit.)
Tuning boosting's n_estimators by hand while ignoring learning_rate. The two are coupled: halve the learning rate and you need roughly double the rounds. Fix lr small, cap rounds high, let early stopping decide.
Rigging the bake-off: tuning your favorite model while opponents run defaults, or giving models different splits/metrics. Same data, same metric, comparable tuning effort — or the table is fiction.
Skipping the linear baseline because "trees always win tabular." On small-n, wide, or nearly-linear data, regularized logistic regression regularly wins — and it is 100× cheaper to serve.
Reading forest probabilities as calibrated. Vote fractions are not honest probabilities; if downstream decisions consume them, calibrate (Day 75 touches this).
Using boosting's training loss to pick rounds. Training loss decreases forever; only the validation curve knows when to stop — that is what early_stopping watches.

Knowledge check

Q1. Random forests fight overfitting primarily by…

Q2. You halve a gradient-boosting model's learning_rate. To recover the same fit quality you should expect to…

Q3. Bagging vs boosting in one line — which is correct?

Go deeper — curated resources

docsscikit-learn User Guide — 1.11 Ensembles: forests & gradient boosting ↗30 min videoStatQuest — Random Forests & Gradient Boost series ↗30 min courseKaggle Learn — Intermediate ML (XGBoost lesson) ↗25 min

If you have a third hour

XGBoost documentation (local install) ↗ — Same boosting mental model with regularized objectives and production tooling. Try replacing HistGradientBoosting in your bake-off locally — the API is nearly identical.

Done means

Bake-off table recorded with fairness audit answered in writing
Learning-rate sweep run; rounds-used pattern explained
Hand-rolled bagging beats mean single tree; missing-ingredient sentence written
bakeoff() and memo committed
Quiz ≥ 2/3

How this connects

← Back: Day 73's unstable, memorizing trees are the raw material — bagging is Day 60's bootstrap plus the law of large numbers, applied to models instead of sample means.

Forward →: Day 76 explains today's magic in bias/variance language. Day 77's competition and Day 83's churn project expect a boosted model in the lineup. On Day 129 the same idea returns for LLMs — routing across several models is ensembling by another name.

Unlocks: D76 Validation, Bias/Variance & Regularization · D77 Week 11 Checkpoint: Tabular Mini-Competition