Decision Trees
- Explain how a tree greedily picks splits by impurity reduction (Gini/entropy)
- Demonstrate — by running it — that an unconstrained tree memorizes the training set
- Control overfitting with max_depth, min_samples_leaf, and cost-complexity pruning
- Read a fitted tree with export_text/plot_tree and explain one prediction path aloud
- State the honest caveats on feature_importances_
| Spaced-rep warm-up: Days 71–72 cards + Day 62 entropy card | 10 min |
| ELI5 + tech read; decision-tree visualizer | 20 min |
| Guided: watch it memorize + read the tree aloud | 37 min |
| Practice: cost-complexity pruning | 20 min |
| Project: overfit_demo.py | 23 min |
| Quiz + flashcards | 10 min |
Builds on: Day 71 — ML framing, splits & baselines · Day 29 — Binary trees & traversal · Day 62 — Entropy as surprise
In Twenty Questions, a good player never opens with "is it a 1997 Honda Civic?" They ask "is it alive?" — the question that splits the world of possibilities most evenly, so either answer eliminates the most candidates. Then, inside the surviving half, they pick the next most-splitting question. Twenty well-chosen questions can pinpoint one thing among a million, because each question does the most possible work at that moment.
A decision tree learns to play Twenty Questions against your data. It examines every feature and every cutoff — "is monthly_charges > 70?" — and picks whichever question best separates the labels into purer piles. Then it recurses into each pile and asks again. Prediction is just walking the questions from the root to a leaf and answering with whatever label dominates there. The catch: given unlimited questions, the tree ends up asking "is it row #4,072?" — perfectly sorting the training data by memorizing it. Real skill in Twenty Questions is asking FEW questions; real skill in trees is knowing when to stop.
Trees matter for three career reasons. They are the building block of the ensembles (Day 74) that still win most tabular problems you will meet in industry — churn, fraud, pricing — which is most customer data. They are the most explainable model family: an FDE can walk a compliance officer through the exact rule path that flagged a transaction, which is sometimes the difference between a deployed model and a rejected one. And "watch the tree memorize" is the cleanest overfitting demo in all of ML — the intuition you will reuse when Day 89's neural nets do the same thing with more drama.
Twenty questions — a tree splits churn data, one question at a time
step 1 / 5A trained tree predicting customer churn. Each internal node asks ONE question chosen to split the data as cleanly as possible (lowest impurity). Leaves hold verdicts.
Guided practice
Watch it memorize
22 min- Paste the starter (breast-cancer dataset again — standalone; run locally if the browser lacks scikit-learn). It sweeps max_depth from 1 to 12 and prints train vs test accuracy for each.
- Run it and find three regimes in the table: underfit (both low, depth 1–2), sweet spot (test peaks), memorize (train hits 1.000 while test stalls or slips). Write down the depth where the gap starts growing.
- Note the depth where train accuracy reaches 1.000 exactly — the tree has a private leaf-path for every hard training sample. Test accuracy did not earn any of that.
- Fix it a second way: keep depth unlimited but set
min_samples_leaf=10. Compare against the best max_depth run. Two different knobs, same medicine — capacity control. - Plot or tabulate the gap (train − test) vs depth: this exact curve shape returns as Day 76's bias-variance picture and Day 89's neural-net loss curves.
Read the tree aloud
15 min- Fit a small tree (
max_depth=3) and print its rules withexport_text(tree, feature_names=...). Read the top split: which feature and threshold did the greedy search choose first? That is the data's best opening question. - Take one test sample and trace its path by hand through the printed rules — answer each question from the sample's feature values until you reach a leaf. Confirm your walk matches
tree.predict. - Print
tree.feature_importances_zipped with feature names, sorted. Then refit with a different random_state and a bootstrap resample of the training data (np.random.default_rng(1).choice(len(X_tr), len(X_tr))as indices) and print importances again. Watch them shuffle — write one sentence on why this instability recommends caution (and foreshadows why forests average many trees).
On your own
Prune it like you mean it
20 minUse DecisionTreeClassifier.cost_complexity_pruning_path(X_tr, y_tr) on the breast-cancer training split to get the ccp_alphas array. Fit one tree per alpha (skip the last, which prunes to a stump), and record each tree's leaf count, train accuracy, and test accuracy.
Goal: (1) produce the table; (2) pick the alpha you would ship and defend the choice in two sentences (test score AND simplicity); (3) compare your pruned tree's leaf count against the unpruned tree's — how many leaves were memorization?
Hints: expect dozens of alphas; iterate with a list comprehension. The shipped tree is rarely the top test scorer — prefer the simplest tree within a whisker of the best (a taste of the one-standard-error rule, formalized on Day 76).
The overfitting demo you will reuse forever
Create overfit_demo.py in your practice repo: a self-contained script that (1) runs the depth sweep on breast-cancer and prints the three-regime table with a MEMORIZING/SWEET-SPOT/UNDERFIT tag per row, (2) prints the depth-3 rules via export_text with a hand-written comment translating the top split into plain English, and (3) ends with a docstring summarizing the lesson in 3 sentences for your future self. This script becomes your standard show-a-stakeholder-what-overfitting-is artifact — FDEs give this exact demo to customers who ask "why can't we just fit the data perfectly?". Commit it.
Common mistakes & misconceptions
- Judging a tree by train accuracy. An unconstrained tree ALWAYS reaches ~100% on train — that number carries zero information about skill.
- Scaling features for trees out of habit. Splits compare feature ≤ threshold; monotonic scaling changes nothing. Save the pipeline complexity for models that need it.
- Reading feature_importances_ as causal truth. They are impurity bookkeeping on train data: biased toward high-cardinality features and arbitrary among correlated ones. Use permutation importance on held-out data for decisions.
- Expecting a regression tree to extrapolate. Leaves predict constants from the training range; ask for a prediction outside it and you get the nearest edge leaf's value, flat forever.
- Believing the greedy tree is the optimal tree. Greedy split selection has no lookahead — a mediocre first split can be locally best. This is fine; ensembles (tomorrow) fix it better than clever single trees.
- Tuning depth on the test set. That silently converts your exam into homework. Tune on a validation split or CV (Day 76); touch test once.
Q1. An unconstrained decision tree reports train accuracy 1.000 and test accuracy 0.71. What happened?
Q2. Which model change does NOT require rescaling or re-encoding numeric features?
Q3. Feature A (a high-cardinality ID-like column) tops feature_importances_. Your best first move is…
Go deeper — curated resources
- Why axis-aligned splits staircase diagonal boundaries — A tree can only cut perpendicular to axes, so a 45° class boundary needs many small rectangular steps. Oblique trees and linear models handle it in one cut — a reason to keep linear baselines around.
- Depth-sweep table produced with the three regimes identified and the gap-growth depth noted
- One prediction traced by hand through export_text rules and verified against predict
- Pruning table built; shipped alpha chosen and defended in writing
- overfit_demo.py committed
- Quiz ≥ 2/3
← Back: The tree structure and root-to-leaf walk are Day 29's binary trees wearing work clothes, and the entropy criterion is Day 62's surprise measure choosing questions. Day 71's train/test discipline is what exposed the memorization.
Forward →: Tomorrow (Day 74) fixes the single tree's instability by averaging hundreds of them — bagging — and by building them sequentially — boosting. Day 76 names today's depth-sweep curve "bias-variance," and Day 80's error analysis starts from the same what-did-it-get-wrong instinct.
Unlocks: D74 Ensembles — Forests & Boosting