PCA & Dimensionality
- Explain the curse of dimensionality and why distance loses meaning in high dimensions
- Describe PCA as finding orthogonal directions of maximum variance, and connect it to Day 52's SVD
- Read an explained-variance curve and choose a component count deliberately
- Use PCA correctly inside a supervised workflow: fit on train only, scale first
- Distinguish PCA-for-visualization from PCA-for-features and state the caveats of each
| Spaced-rep warm-up: Day 52 SVD cards + Day 78 deck | 10 min |
| ELI5 + tech read; pca-project visualizer | 20 min |
| Guided: squash the digits + reconstruction accounting | 37 min |
| Practice: does PCA help the model? | 20 min |
| Project: toolkit lens + pca_lab.md | 23 min |
| Quiz + flashcards | 10 min |
Builds on: Day 52 — Eigenvectors & SVD intuition · Day 78 — Clustering & scaling discipline · Day 64 — NumPy arrays & axes
Photograph a galloping horse. From the side, one photo tells you almost everything — legs extended, mane flying. From directly above, the same horse is a brown smudge; from head-on, a bobbing oval. Same 3-D animal, but the ANGLE decides how much of its story survives the flattening into 2-D. There exists a best angle: the one where the horse's shape spreads out the most, so the least information is lost when depth is discarded.
PCA finds that best angle for data. Your dataset lives in 30 or 64 dimensions — unphotographable — but its interesting variation often sprawls along just a few directions, the way the horse's action lives in the side view. PCA rotates the axes to point along the directions where the data spreads widest (the first principal component is the widest, the second is the widest at right angles to it, and so on), then lets you keep the top few and drop the rest. The explained-variance readout tells you exactly how much of the story each angle captures — "these 2 directions hold 85% of the spread" — so the flattening is a measured trade, not a guess.
PCA earns its keep three ways in this program and your career. Practically: it is the standard microscope for anything high-dimensional — tomorrow you would struggle to see Day 78's clusters in 64 dimensions, and on Day 92 you will use exactly today's code to look at LLM embedding spaces. Statistically: it tames the curse of dimensionality that quietly breaks distance-based methods (kNN, k-means) as features multiply. And conceptually: the compress-to-what-matters idea returns as the intuition behind embeddings themselves and LoRA's low-rank adapters on Day 128. Interviewers love "when would PCA hurt?" — you will have an answer.
The best camera angle — projecting 2D onto its principal axis
step 1 / 5Height vs shoe size for 9 people. Two numbers each — but the cloud is basically a diagonal line: the two features move together.
Guided practice
Squash the digits: 64 dimensions → 2
22 min- Paste the starter. load_digits gives 1,797 handwritten digits as 8×8 images = 64 features. Standalone; run locally if needed.
- Run it. Read the cumulative explained-variance printout: how many components hold 50%? 90%? Write both numbers down — 64 pixels were never 64 independent dimensions.
- Look at the 2-D projection printout: per-digit centroid positions in PC space. Digits 0 and 1 should sit far apart; 4 and 9 close (they share strokes). If you have matplotlib, scatter the projection colored by digit — the clusters are visible to the naked eye in a 2-D shadow of 64-D space.
- Note the axis labels you MUST always include: PC1 holds ~X% of variance, PC2 ~Y%. A 2-D picture holding 28% of the variance is a sketch, not the territory — say so on the plot.
Reconstruction: watch the variance you paid
15 min- For k in [2, 10, 20, 40, 64], fit PCA(k) on the scaled digits, transform, then
inverse_transformback to 64-D. Compute the mean squared reconstruction error against the input. - Tabulate k, cumulative variance kept, and reconstruction error. Confirm the accounting identity: error shrinks exactly as kept-variance grows, hitting ~0 at k=64.
- If you have matplotlib, render one digit's original vs its k=10 reconstruction as 8×8 images — recognizably the same digit from 10 numbers instead of 64. That compression-preserving-identity is the intuition to keep for embeddings (Day 92).
- Two-sentence journal: what did PCA throw away first, and why is that usually (but not always!) the right thing to discard?
On your own
Does PCA help the model? Measure, don't assume
20 minRun an honest experiment on the digits with your Day 76 tools: 5-fold CV accuracy for (a) LogisticRegression on all 64 scaled features, (b) a Pipeline of StandardScaler → PCA(n_components=0.90) → LogisticRegression, (c) same but n_components=2.
Goal: (1) report mean ± std for all three; (2) note the feature counts (64 vs ~how many vs 2) and what accuracy each bought; (3) answer in writing: did PCA-for-features help, hurt, or wash here — and what did the 2-component version prove about "variance ≠ relevance"? (4) name one scenario from the tech section where you would EXPECT (b) to beat (a).
Hints: build the PCA inside the Pipeline so folds don't leak. Expect (a) ≈ (b) on this dataset with (b) using far fewer features — a compression win, not an accuracy win. (c) should drop hard: 28% of variance is not 28% of the signal.
The projection lens for your toolkit
Add pca_summary(X, variance_targets=(0.5, 0.9, 0.95)) to ml_toolkit.py: scales, fits PCA, and returns the components needed per variance target plus the top-5 explained-variance ratios — your instant "how flat is this data?" probe. Add project_2d(X, labels=None) returning the scaled 2-D projection with the two variance percentages, printing the axis-label warning string. Then create pca_lab.md recording today's numbers: the digits' 50/90/95% component counts, the reconstruction table, and your practice experiment's three CV scores with your verdict paragraph. Commit both; project_2d gets called again on Day 92 to look at real embeddings.
Common mistakes & misconceptions
- Running PCA on unscaled data. PCA maximizes variance, and variance has units — the largest-scaled feature becomes PC1 by default. Standardize first unless features share units on purpose.
- Fitting PCA on the full dataset before a supervised split or CV. Components are learned parameters; fitting on all rows leaks test structure. Pipeline it like any transformer.
- Reading a 2-D PCA plot as the truth. If PC1+PC2 hold 30% of variance, 70% of the structure is invisible. Always print the percentages on the axes.
- Assuming top variance = top relevance. PCA never saw the labels; the discriminative signal can hide in component 12. Validate downstream accuracy, never just the scree plot.
- Interpreting component signs or exact loadings too confidently. Signs are arbitrary and components are mixtures; treat loadings as directional hints, not named factors.
- Reaching for PCA before a tree ensemble. Trees do axis-aligned feature selection natively and lose interpretability behind PCA; the win zone is distance-based and linear models on wide, collinear data.
Q1. Why must data be standardized before PCA (in general)?
Q2. Your 2-D PCA plot shows PC1 = 18%, PC2 = 11% explained variance and no visible clusters. The safest conclusion is…
Q3. PCA down to k components, then inverse_transform. The reconstruction error equals…
Go deeper — curated resources
- t-SNE and UMAP vs PCA — Nonlinear neighbor-embedding methods make prettier cluster plots but distort global geometry and have fiddly knobs (perplexity). Rule: PCA first for honesty and speed; t-SNE/UMAP for presentation, never for downstream features. You will meet them on Day 92.
- Component counts for 50/90/95% variance recorded for digits
- Reconstruction table confirms the variance-error accounting
- Three-way CV experiment run; verdict paragraph written
- pca_summary and project_2d committed
- Quiz ≥ 2/3
← Back: This is Day 52's SVD cashing its promissory note: eigen-directions of the covariance are the directions that only stretch, and low-rank approximation became a scree plot. Day 78's scaling discipline carried straight over — PCA is variance-chasing the way k-means is distance-chasing.
Forward →: On Day 92 project_2d becomes your window into embedding space, and the compression intuition underpins why embeddings work at all. Day 128's LoRA is the same low-rank bet applied to weight updates: the important changes live in few directions.
Unlocks: D92 Embeddings — Meaning as Geometry