Day 52 · The grain of the wood

Rank, Eigenvectors & SVD Intuition

You will be able to

Define rank as the true dimensionality of what a matrix produces, and detect it numerically
Explain eigenvectors as directions a machine only stretches, and find one by power iteration
Describe SVD as rotate → stretch → rotate and read the singular values as importance scores
Compress a matrix with a truncated SVD and measure the error-vs-rank trade-off
Connect low-rank structure to LoRA (Day 128) and PCA (Day 79) in one paragraph each

Today's ~125 minutes

Spaced-rep warm-up: due flashcards (Days 50–51)	10 min
ELI5 + tech read + 3B1B eigenvectors chapter	25 min
Guided: power iteration + SVD compression	45 min
Practice: the rank detective	20 min
Project: low-rank lab report	15 min
Quiz + flashcards	10 min

Builds on: Day 51 — Matrices as transformations · Day 50 — Dot products & norms

The analogy

🧸 ELI5⚙ Tech

Run your hand across a plank of wood. It feels totally different along the grain versus across it — the wood has built-in directions, and everything you do to it (splitting, sanding, bending) works with or against them. Matrix machines have grain too. Most arrows you feed a machine come out rotated AND stretched — but a few special arrows come out merely stretched, still pointing the way they went in. Those are the eigenvectors — the machine's grain — and the stretch factors are the eigenvalues. Push anything through the machine repeatedly and it gets combed toward the strongest grain direction, like sanding always ends up following the wood.

Now the deeper cut, the SVD: every machine — every matrix, no exceptions — is secretly three simple moves glued together: a rotation, then a pure stretch along perpendicular axes (by amounts called singular values), then another rotation. The singular values are an importance ranking of the machine's directions. And here is the trick that pays your salary later: if only a few singular values are large, the machine is impersonating a much simpler machine — you can throw away the small ones and keep almost all the behavior at a fraction of the size. That is a low-rank approximation, and it is precisely the bet LoRA makes when it fine-tunes a giant model with tiny sticky-note matrices on Day 128.

Why this matters on the job

Low-rank structure is one of the most monetizable ideas in modern AI. LoRA fine-tunes billion-parameter models by learning updates constrained to rank r ≈ 8–64 — millions of times smaller than the weight matrices they adjust — and it works because useful weight *changes* empirically have low rank; when you run a LoRA lab on Day 128, "rank" is a dial you will set with today's intuition. PCA (Day 79) is SVD applied to data, powering the 2D maps you will draw of embedding spaces (Day 92). Recommender systems, image compression, and attention-matrix analysis all speak this language: which directions matter, and how many are there really?

Guided practice

guided 1

Find the grain: power iteration by hand

20 min

Run part 1: the matrix A stretches direction [1,1] by 3 and direction [1,−1] by 0.5 (it is built from those eigenvectors). Feed it a random vector and apply A repeatedly, normalizing each step. Watch the printed vector swing toward [0.707, 0.707] within ~8 iterations — the strongest grain wins.
Check against np.linalg.eig: confirm eigenvalues ≈ 3 and 0.5, eigenvectors ≈ [1,1]/√2 and [1,−1]/√2.
Verify the definition directly: compute A @ v for the found v and confirm it equals 3v (parallel, stretched, not rotated) — then do the same for a NON-eigenvector and see the direction change.
Break it: replace A with a 90° rotation matrix and re-run power iteration. It never settles — a rotation has no real grain. Note WHY in one line (nothing keeps its direction under rotation).

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

SVD anatomy + the compression deal

25 min

Part 1 decomposes a 3×2 matrix with np.linalg.svd and verifies the anatomy: U and V have orthonormal columns (rotations), Σ is a non-negative stretch, and U @ diag(σ) @ Vᵀ rebuilds A exactly.
Part 2 builds a 60×60 "image" (a smiley-like pattern of blocks) that is secretly low-rank, adds noise, and prints the first ten singular values. Read the cliff: a few large σ (the pattern), then a floor of small ones (the noise). Effective rank = the count above the cliff.
Reconstruct at ranks k = 1, 2, 4, 8 and print the relative error and compression ratio for each. Find the k where the error stops improving meaningfully — you have separated structure from noise.
Write the LoRA sentence in your notes, filling the blanks from YOUR run: "keeping k=…, I stored …% of the numbers and kept …% of the structure; LoRA bets weight UPDATES are compressible exactly like this."

🐍 python — editable, runs in your browser

import numpy as np

rng = np.random.default_rng(7)

# --- 1. anatomy check on a small matrix
A = np.array([[3.0, 1.0], [1.0, 3.0], [0.0, 2.0]])
U, s, Vt = np.linalg.svd(A, full_matrices=False)
print("singular values:", s.round(3))
print("U columns orthonormal:", np.allclose(U.T @ U, np.eye(2)))
print("V columns orthonormal:", np.allclose(Vt @ Vt.T, np.eye(2)))
print("rebuild exact:", np.allclose(U @ np.diag(s) @ Vt, A))

# --- 2. a low-rank picture drowned in noise
img = np.zeros((60, 60))
img[10:20, 15:25] = 1.0          # left eye
img[10:20, 35:45] = 1.0          # right eye
img[38:44, 12:48] = 1.0          # mouth
img[30:34, 28:32] = 0.5          # nose
noisy = img + rng.normal(0, 0.08, img.shape)

U, s, Vt = np.linalg.svd(noisy)
print("\nfirst 10 singular values:", s[:10].round(2))

# --- 3. truncate: keep the k most important rank-1 layers
total = np.linalg.norm(noisy)
for k in (1, 2, 4, 8):
    approx = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]
    err = np.linalg.norm(noisy - approx) / total
    stored = k * (60 + 60 + 1)
    print(f"rank {k}: rel error {err:.3f}   "
          f"stored {stored}/{60*60} numbers ({stored/3600:.1%})")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

The rank detective

20 min

You get three mystery 50×50 matrices (build them per the spec, then pretend you did not): M1 = outer product of two random vectors; M2 = sum of three outer products; M3 = M2 + tiny noise (scale 1e-6). Using ONLY singular values (no np.linalg.matrix_rank at first):

(1) print each spectrum's first 8 values and state the exact or effective rank; (2) for M3, choose and justify a tolerance that recovers "rank 3" despite 50 nonzero σ; (3) verify with np.linalg.matrix_rank and its tol argument; (4) answer in two sentences: a weight-update matrix from fine-tuning shows σ = [42.1, 18.3, 9.7, 0.3, 0.29, 0.28, …]. What LoRA rank would capture it, and what would you lose at r=2?

Hints (only if stuck): an outer product u vᵀ is exactly rank 1 — the machine A = u vᵀ sends every input somewhere on the line through u; sums of k outer products have rank ≤ k.

Ship before you stop

Low-rank lab report

Create svd_lab.py + lowrank_notes.md. The script: the smiley compression sweep (ranks 1–10) printing error and storage tables, plus your rank-detective solutions. The notes: (1) "the grain of the wood" retold in your own words with the power-iteration evidence; (2) the SVD anatomy (rotate-stretch-rotate) with your orthonormality checks as proof; (3) the compression table and the k you chose, with the structure-vs-noise argument; (4) two forward-looking paragraphs written to your future self — "Dear Day 128: LoRA constrains ΔW = B·A at rank r because…" and "Dear Day 79: PCA is SVD on centered data; the top singular directions are the best camera angles because…". These two paragraphs get re-read on those days — write them well. Commit both files.

Rubric — check what you completed (0/6)

Compression sweep table covers ranks 1–10 with error and storage columnsRank detective identifies 1, 3, and effective-3 with a justified tolerancePower-iteration evidence (convergence to [1,1]/√2) recordedOrthonormality and exact-rebuild checks pass in the scriptBoth "Dear Day" paragraphs written and technically correctCommitted with a descriptive message

Common mistakes & misconceptions

Conflating eigenvectors and singular vectors. Eigen: square matrices, input direction preserved (Av = λv), possibly complex. SVD: any matrix, two DIFFERENT orthonormal direction sets (in and out), always real and non-negative σ. Related, not identical.
Expecting exact zeros to reveal rank in real data. Noise makes every σ nonzero; effective rank = counting above a tolerance you must choose and justify. The cliff in the spectrum is your friend.
Thinking low-rank approximation is lossy compression you could beat with cleverness. Eckart–Young: SVD truncation is provably the best rank-k approximation in Frobenius norm. The only question is choosing k.
Believing every matrix has real eigenvectors. Rotations stretch nothing — their eigenvalues are complex. Power iteration spinning forever is the symptom you saw.
Missing why LoRA is cheap: it does not compress the frozen model, it constrains the UPDATE to rank r — storage and training cost scale with r(m+n), not m·n. The bet is that adaptation needs few directions.
Reading σ magnitudes without normalizing context. Singular values scale with the data; compare them to each other (ratios, cumulative energy), not to constants remembered from another matrix.

Knowledge check

Q1. v is an eigenvector of A with eigenvalue 3. What is A @ (A @ v)?

Q2. A 1000×1000 matrix has singular values [95, 40, 12, 0.02, 0.01, …]. The best summary is:

Q3. LoRA fine-tunes by learning ΔW = B·A with B ∈ ℝ^(m×r), A ∈ ℝ^(r×n), r ≈ 8. What is the underlying mathematical bet?

Go deeper — curated resources

video3Blue1Brown — Essence of Linear Algebra, ch. 14 (eigenvectors & eigenvalues) ↗20 min courseMIT OCW 18.06 — lectures 21–22 (eigenvalues) & 29 (SVD) ↗40 min bookMathematics for Machine Learning — ch. 4 (matrix decompositions) ↗30 min bookImmersive Linear Algebra — interactive decomposition figures ↗15 min

If you have a third hour

PCA = SVD on centered data — Center the data matrix, take its SVD: right singular vectors are the principal components, σᵢ² the explained variances. Day 79 does this properly; peek now if curious.
The LoRA paper (Hu et al. 2021) ↗ — Section 4.1 states the low-rank hypothesis in two sentences you can now read. Save the full read for Day 128.

Done means

Power iteration converges to the known eigenvector; rotation counterexample noted
SVD anatomy checks pass; compression sweep table recorded
Rank detective solved with a justified tolerance
Both "Dear Day 128 / Day 79" paragraphs committed
Quiz ≥ 2/3

How this connects

← Back: Day 51's flattener was a rank-deficient machine before you had the word; its parallel columns were dependent columns. Power iteration is just repeated Day 51 composition, and every σᵢuᵢvᵢᵀ layer is built from Day 50 outer products of unit vectors.

Forward →: Day 79 runs SVD on centered data and calls it PCA — your "best camera angle" for visualizing Day 92's embedding spaces. Day 128 makes the low-rank bet operational: choosing LoRA's r IS choosing a truncation rank for the adaptation. Tomorrow pivots from the shape of transformations to the shape of change: derivatives.

Unlocks: D79 PCA & Dimensionality · D128 Fine-Tuning II — LoRA Lab