Logistic Regression & Losses
- Explain how the sigmoid turns a linear score into a probability and what the decision boundary is
- Show that log-loss is exactly the cross-entropy from Day 62, and why MSE is the wrong loss for probabilities
- Use predict_proba and a chosen threshold instead of blindly trusting predict
- Handle class imbalance with class_weight and threshold moves, and explain the difference
- Describe what the C parameter regularizes, at first-taste level
| Spaced-rep warm-up: Day 62 cross-entropy cards + Day 71 deck | 10 min |
| ELI5 + tech read; sigmoid-boundary visualizer | 20 min |
| Guided: probabilities under the microscope + threshold sweep | 38 min |
| Practice: the stalled gradient | 18 min |
| Project: evaluate_classification in ml_toolkit.py | 24 min |
| Quiz + flashcards | 10 min |
Builds on: Day 71 — ML framing & the estimator API · Day 62 — Entropy & cross-entropy · Day 69 — Scaling & pipelines
Two doctors look at the same scan. Both say "not cancer." But one means "99.9% sure — go home" and the other means "51% sure — I flipped a mental coin." Same answer, wildly different information. You would want the second patient sent for more tests. A classifier that only outputs labels is the coin-flip doctor with a confident voice: it hides how sure it was, and hiding that throws away the most decision-relevant number in the system.
Logistic regression is the doctor who says the percentage out loud. It computes the same weighted score as yesterday's straight line, then squashes it through an S-shaped curve (the sigmoid) so the output lands between 0 and 1 — a probability. The "answer" is just you choosing a cutoff: above 50%, call it positive. But that 50% is YOUR policy choice, not the model's. A fraud team might act at 5% suspicion; a spam filter might wait for 99%. Today you learn to keep the confidence and choose the cutoff deliberately.
Probabilities, not labels, are what production systems act on: route the risky transaction to review above 0.05, auto-approve below. Thresholding is a business decision an FDE negotiates with the customer ("what does a false alarm cost you? a miss?") — Day 75 turns that into metric choice. And the loss you meet today, cross-entropy, is THE loss: every neural network you train from Day 85 on, and every LLM ever pretrained, minimizes exactly this quantity. Understanding it once here pays compound interest for the rest of the program.
Confidence, not just answers — the sigmoid and the moving threshold
step 1 / 5A linear model outputs any score from −∞ to +∞. The sigmoid squashes that score into a probability between 0 and 1 — confidence, not just a verdict.
Guided practice
Probabilities under the microscope
20 min- Paste the starter. It trains logistic regression (inside a scaling pipeline) on the built-in breast-cancer dataset — runs standalone; if scikit-learn is missing in the browser, run locally.
- Print the first 8 test rows: predicted probability, predicted label, true label. Find a case where the model was right but unsure (p between 0.4 and 0.7). Would you ship an auto-decision on that row?
- Compute log-loss and accuracy on the test set. Now build a deliberately overconfident copy of the probabilities with
np.clip(proba, 0.001, 0.999)pushed to extremes:np.where(proba > 0.5, 0.999, 0.001). Accuracy is unchanged — but recompute log-loss. Explain the jump in one sentence. - Hand-verify one sample's log-loss term with
-np.log(p)(true class 1) or-np.log(1-p)(true class 0) and match it against sklearn's average.
Move the threshold, move the world
18 min- Simulate an imbalanced problem:
make_classification(n_samples=4000, weights=[0.95, 0.05], random_state=0)— 5% positives, like fraud. - Train the same scaled logistic pipeline. At the default 0.5 threshold, count: how many true positives did it catch, how many did it miss? Use plain NumPy comparisons —
((proba >= t) & (y_test == 1)).sum()— no metric imports yet (that is Day 75's job). - Sweep thresholds t in [0.9, 0.7, 0.5, 0.3, 0.1]. For each, print caught positives, missed positives, and false alarms. Watch the trade: lower t catches more real positives at the price of more false alarms.
- Retrain with
class_weight="balanced"at fixed t = 0.5 and compare to the threshold move. Write two sentences: which lever changed the model, and which changed only the policy?
On your own
The stalled gradient, felt by hand
18 minProve to yourself why cross-entropy beats MSE for classification. For a single true-positive sample (y = 1), compute both losses and their gradients with respect to p at p = 0.9, 0.5, 0.1, 0.01: cross-entropy loss −log(p) with gradient −1/p, and squared loss (1−p)² with gradient −2(1−p).
Goal: (1) tabulate loss and |gradient| for both at all four p values (NumPy or by hand); (2) identify where the model is most catastrophically wrong (p = 0.01) and compare the gradient magnitudes there; (3) write the conclusion in two sentences.
Hints: at p = 0.01 the cross-entropy gradient is 100 — a shove proportional to the wrongness. Note the squared-loss gradient there stays under 2, and in a real network the sigmoid's own slope shrinks it much further toward zero — the stall.
Extend the toolkit to classification
Add evaluate_classification(model, X_train, X_test, y_train, y_test, threshold=0.5) to ml_toolkit.py. It must fit the model, fit a DummyClassifier(strategy="most_frequent") baseline on the same split, and return accuracy and log-loss for both, plus positive-catch and false-alarm counts at the given threshold computed via predict_proba (raw NumPy counting — you will upgrade these to named metrics on Day 75). Add a threshold parameter demo in __main__ on the imbalanced dataset from guided exercise 2 showing the baseline's deceptive accuracy. Commit.
Common mistakes & misconceptions
- Trusting predict and never looking at predict_proba. The 0.5 cutoff is a default policy, not a law — most business problems need a different threshold, chosen from costs.
- Confusing log-loss with accuracy. Accuracy only sees which side of the threshold; log-loss also punishes miscalibrated confidence. Two models with equal accuracy can have wildly different log-loss.
- Using MSE as a classification loss. It goes non-convex through the sigmoid and its gradient vanishes exactly where the model is confidently wrong; cross-entropy's gradient (p − y) does not stall.
- Fixing imbalance ONLY by reweighting when a threshold move suffices — or thresholding when the boundary itself is wrong. Know which lever you pulled and why; reweighting also distorts probability calibration.
- Forgetting to scale features. Unscaled features slow the solver (convergence warnings) and make C's penalty hit large-unit features unevenly. Pipeline + StandardScaler is the reflex.
- Reading "logistic REGRESSION" and using it on continuous targets. The name is historical; it is a classifier that regresses log-odds.
Q1. A model predicts p = 0.02 for a sample whose true label is 1. Compared with predicting p = 0.4, its log-loss contribution is…
Q2. Where is the decision boundary of a logistic regression model?
Q3. For a 5%-positive fraud problem, lowering the decision threshold from 0.5 to 0.1 will typically…
Go deeper — curated resources
- Calibration: when 0.7 should mean 70% — A model is calibrated if events predicted at p happen a fraction p of the time. Trees and boosted models are often miscalibrated; sklearn's CalibratedClassifierCV fixes it. Tasted properly on Day 75.
- Overconfidence experiment run: same accuracy, exploded log-loss, explained in one sentence
- Threshold sweep table produced; two-sentence answer distinguishing threshold vs class_weight
- Gradient comparison table from practice completed with written conclusion
- ml_toolkit.py extended and committed
- Quiz ≥ 2/3
← Back: Day 62 defined cross-entropy as surprise-weighted cost — today it became the training loss, paying off exactly as promised. The boundary w·x + b = 0 is Day 50's dot product drawing a line in space.
Forward →: Day 75 gives proper names (precision, recall) to the counts you tallied by hand today. Day 76 explains the C parameter fully. From Day 85 on, every neural net — and every LLM in pretraining — minimizes this same cross-entropy, just with billions of parameters.
Unlocks: D75 Evaluation Metrics · D85 Neurons & Forward Pass · D93 word2vec Lab