Day 51 · Machines that move space

Matrices & Transformations

You will be able to

Interpret a matrix as a transformation of space and read its columns as where the basis vectors land
Apply rotation, scaling, and shear matrices to point clouds and predict the result before running
Explain why composing transformations is matrix multiplication and why order matters
Describe identity and inverse as "do nothing" and "undo," and name when no inverse exists
Time vectorized matmul against Python loops and explain why GPUs are matmul machines

Today's ~125 minutes

Spaced-rep warm-up: due flashcards (Day 50 + Week 7)	10 min
ELI5 + tech read + 3B1B ch. 3–5, matrix-transform visualizer	25 min
Guided: point-cloud machines + layer-and-timing lab	45 min
Practice: design the machine	20 min
Project: gallery + notes	15 min
Quiz + flashcards	10 min

Builds on: Day 50 — Vectors & dot products · Day 22 — Big O — counting the multiplies

The analogy

🧸 ELI5⚙ Tech

A matrix is not a spreadsheet of numbers — it is a machine that moves space. Feed it any arrow (vector), and it outputs a moved arrow. One machine rotates everything 90°; another stretches everything horizontally; another squashes the whole plane onto a line. And here is the beautiful secret that makes them readable: to know everything a machine does, you only need to watch what it does to two test arrows — "one step right" and "one step up." The machine's columns ARE those answers. Columns [0,1] and [−1,0]? Right went up, up went left: it is a rotation. You can read the machine's soul off its columns.

Chaining machines — rotate, THEN stretch — is itself a single machine, and computing it is what matrix multiplication *is*. That is why order matters: rotating then stretching is a different trip than stretching then rotating, just like putting on socks then shoes differs from shoes then socks. The identity matrix is the machine that touches nothing; an inverse is the machine that exactly undoes another — and machines that flatten space (squash 2D onto a line) cannot be undone, because flattening destroys information. Hold that thought: it returns tomorrow as "rank."

Why this matters on the job

A neural network layer is literally a matrix machine: output = W @ x + b, where W is learned (Day 85). A transformer forward pass is a long chain of these machines — and attention (Day 94) computes its scores as one big matmul of queries against keys, exactly the "every row shadow-tests every column" picture you build today. Matmul is such an overwhelming fraction of AI compute that an entire hardware industry (GPUs, TPUs) exists to do it fast — which is why "can you vectorize this?" is a real performance question you will face in every pipeline you ship, and why today's loop-vs-matmul timing is a number worth remembering.

Watch it happen

Machines that move space — read a matrix by where it sends the unit vectors

step 1 / 6

	A·e1	A·e2
x	2	0
y	0	1

The secret to reading any matrix: its COLUMNS are the destinations of the unit vectors. Matrix A sends e1=(1,0) to (2,0) and e2=(0,1) to (0,1) — a 2× horizontal stretch.A = [[2, 0], [0, 1]] — columns = where the axes land

🎯 Predict modespeed1×

Guided practice

guided 1

Machines that move a point cloud

25 min

Run part 1: a unit square (four corners + center) is pushed through four machines: rotation(45°), scale(2, 0.5), shear, and a "flattener." For each, the corners are printed before/after. BEFORE running each, read the matrix columns aloud and predict where [1,0] and [0,1] land — then check.
Part 2 composes: R @ S versus S @ R applied to the same square. Confirm the corner coordinates differ — order matters. Write one sentence: which corner proves it?
Part 3: undo. Apply np.linalg.inv(R @ S) to the transformed square and confirm you recover the original corners (to ~1e-15). Then try to invert the flattener and watch NumPy raise LinAlgError — flattening is irreversible: two corners already landed on the same line.
In your notes: the columns-tell-all rule in your own words, plus why the flattener's columns give it away (they are parallel — both point along one line).

🐍 python — editable, runs in your browser

import numpy as np

square = np.array([[0, 0], [1, 0], [1, 1], [0, 1], [0.5, 0.5]], float).T
# shape (2, 5): each COLUMN is a point, so A @ square moves all points at once

t = np.radians(45)
machines = {
    "rotate 45":  np.array([[np.cos(t), -np.sin(t)],
                            [np.sin(t),  np.cos(t)]]),
    "scale 2,.5": np.array([[2.0, 0.0],
                            [0.0, 0.5]]),
    "shear":      np.array([[1.0, 1.0],
                            [0.0, 1.0]]),
    "flattener":  np.array([[1.0, 2.0],
                            [0.5, 1.0]]),   # columns are parallel!
}

for name, A in machines.items():
    moved = A @ square
    print(f"\n{name}: columns -> {A[:,0].round(2)} and {A[:,1].round(2)}")
    print("  before:", square.T.round(2).tolist())
    print("  after: ", moved.T.round(2).tolist())

# --- 2. order matters
R, S = machines["rotate 45"], machines["scale 2,.5"]
print("\nR@S corner (1,1):", (R @ S @ square)[:, 2].round(3))
print("S@R corner (1,1):", (S @ R @ square)[:, 2].round(3))

# --- 3. undo, and the machine that cannot be undone
restored = np.linalg.inv(R @ S) @ (R @ S @ square)
print("\nmax restore error:", np.abs(restored - square).max())
try:
    np.linalg.inv(machines["flattener"])
except np.linalg.LinAlgError as e:
    print("flattener inverse:", e)

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

A layer is a matrix; a GPU is a matmul furnace

20 min

Part 1 builds a tiny "sentiment layer" by hand: W has two rows — a "positive direction" and a "negative direction" in the 4-dim doc space from Day 50. Push the whole 5-doc matrix through it with ONE matmul and read the two scores per doc. Each output is a dot-product alignment with one learned-direction-to-be — this is a dense layer before training.
Confirm the shapes narrate the story: (5,4) @ (4,2) → (5,2). Five docs, two features out.
Part 2 times a (500×500)@(500×500) matmul as triple Python loop vs np.matmul. Record the ratio (expect 3–4 orders of magnitude).
Count the multiplies: 500³ = 1.25×10⁸ — and note that NumPy did them in milliseconds on a CPU. A modern GPU does tens of TERAFLOPs: this operation is what AI hardware is for. Write the measured ratio in your notes; you will quote it on Day 64.

🐍 python — editable, runs in your browser

import numpy as np, time

# --- 1. docs from Day 50: [animal, food, tech, sports]
docs = np.array([
    [0.9, 0.1, 0.0, 0.1],   # cats chase mice
    [0.0, 0.9, 0.0, 0.0],   # best pizza in town
    [0.0, 0.0, 0.9, 0.1],   # new GPU benchmarks
    [0.0, 0.1, 0.1, 0.9],   # match went to penalties
    [0.8, 0.0, 0.0, 0.5],   # dogs love tennis balls
])
W = np.array([
    [0.7, 0.7, 0.0, 0.3],   # row 0: a "cozy topics" direction
    [0.0, 0.0, 0.9, 0.6],   # row 1: a "competitive tech" direction
])
scores = docs @ W.T          # (5,4)@(4,2) -> (5,2): all docs, both features
print("scores (cozy, competitive-tech):\n", scores.round(2))

# --- 2. loops vs matmul
n = 500
A = np.random.default_rng(0).normal(size=(n, n))
B = np.random.default_rng(1).normal(size=(n, n))

t0 = time.perf_counter()
C_loop = np.zeros((n, n))
for i in range(n):
    for j in range(n):
        s = 0.0
        for k in range(n):
            s += A[i, k] * B[k, j]
        C_loop[i, j] = s
loop_s = time.perf_counter() - t0

t0 = time.perf_counter()
C_fast = A @ B
fast_s = time.perf_counter() - t0

print(f"\nloops : {loop_s:8.2f} s")
print(f"matmul: {fast_s*1000:8.2f} ms   ratio ~ {loop_s/fast_s:,.0f}x")
print("same answer:", np.allclose(C_loop, C_fast))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Design the machine

20 min

Reverse the skill: instead of reading matrices, write them. Build each 2×2 machine from the columns-tell-all rule, apply it to the unit square, and verify the printed corners match your intent: (1) rotate 90° counterclockwise; (2) reflect across the y-axis; (3) project everything onto the x-axis (a deliberate flattener); (4) rotate 30° then scale ×3 — as ONE matrix built by multiplication; (5) the machine that undoes (4).

Constraints: construct every matrix by reasoning about where [1,0] and [0,1] must land — no formula lookup; verify (5) with an allclose check against the original square; state which of the five machines has no inverse and how you know from its columns alone.

Hints (only if stuck): reflection sends [1,0]→[−1,0] and keeps [0,1]; projection sends both basis vectors onto the x-axis — parallel columns = flattened = no inverse.

Ship before you stop

Transformation gallery + layer notes

Create matrix_lab.py + matrix_notes.md in your practice repo. The script: your five practice machines plus the guided demos, each printing before/after corner tables with a one-line docstring naming the transformation. The notes file: (1) the columns-tell-all rule with your own 2×2 worked example; (2) the composition section — R@S vs S@R corner evidence and the socks-then-shoes sentence; (3) the invertibility section — which machines undo, which flatten, and how columns betray a flattener; (4) the timing table (loops vs matmul, the measured ratio) under the heading "why GPUs exist"; (5) five sentences titled "a neural layer is a matrix" mapping Y = X @ W.T + b onto today's machines — to be quoted back at you on Day 85. Commit both.

Rubric — check what you completed (0/6)

All five designed machines verified against predicted corners with allcloseThe undo machine restores the square to ~1e-12 errorComposition order shown to matter with specific corner coordinates as evidenceFlattener identified from parallel columns, with the no-inverse explanationTiming table records the measured loop/matmul ratioCommitted with a descriptive message

Common mistakes & misconceptions

Memorizing matmul mechanics without the transformation picture. Row-times-column is HOW; "composition of space-moving machines" is WHY. The picture is what transfers to attention and layers.
Assuming AB = BA. Matrix multiplication is not commutative — rotate-then-scale differs from scale-then-rotate, and you proved it with corner coordinates. It IS associative, which is a different (and useful) property.
Reading AB as "A first, then B." Applied to a vector, (AB)x = A(Bx): B acts first. Read compositions right to left, like function application.
Thinking every square matrix has an inverse. Flatteners (determinant 0, parallel/dependent columns) destroy information; no machine can restore what two inputs merged into one.
Computing explicit inverses in numerical code. Solve the linear system instead (np.linalg.solve) — cheaper and more stable. The inverse is a concept for your head, not usually a matrix for your RAM.
Writing loops where a matmul exists. The 1000×-plus ratio you measured is the argument; shaping problems AS matrix products is the core NumPy (Day 64) and PyTorch (Day 87) skill.

Knowledge check

Q1. The columns of a 2×2 matrix are [0, 1] and [−1, 0]. What does this machine do?

Q2. Why does a matrix with determinant 0 have no inverse?

Q3. X is a batch of 1,000 vectors (1000×768) and W.T is 768×512. What does X @ W.T compute, and why do we care that it is ONE operation?

Go deeper — curated resources

video3Blue1Brown — Essence of Linear Algebra, chapters 3–5 (transformations, composition) ↗35 min courseMIT OCW 18.06 — lecture 1–3 (the geometry of linear equations, multiplication) ↗30 min bookImmersive Linear Algebra — ch. 6 (the matrix, interactive) ↗20 min bookMathematics for Machine Learning — ch. 2 (linear algebra) ↗25 min

If you have a third hour

Determinant as area scaling — det(A) is the factor by which the machine scales areas (volumes in 3D); negative means orientation flips; 0 means flattened. 3B1B chapter 6 makes it visual in ten minutes.

Done means

All guided machines predicted-then-verified via corner tables
Five designed machines pass allclose checks; undo machine works
Loop-vs-matmul ratio measured and recorded
"A neural layer is a matrix" paragraph committed
Quiz ≥ 2/3

How this connects

← Back: Every entry of a matmul is a Day 50 dot product — a matrix-vector product is all rows shadow-testing one vector at once. The loop-vs-vectorized timing repeats Day 22's lesson (count the operations) at the hardware level.

Forward →: Tomorrow asks what a machine does to space beyond moving it: the directions it merely stretches (eigenvectors) and its rotate-stretch-rotate anatomy (SVD) — the key to LoRA on Day 128. On Day 85, Y = X@W.T + b becomes a trained layer; on Day 94, attention scores are Q @ K.T — a matmul you can already read.

Unlocks: D52 Rank, Eigenvectors & SVD Intuition · D64 NumPy in Anger · D85 Neurons & Forward Pass