Day 50 · Arrows and shadows

Vectors & Dot Products

You will be able to

Describe a vector both ways: a list of numbers and an arrow with length and direction
Compute norms and interpret the dot product as alignment between directions
Compute cosine similarity and explain why it ignores vector length
Show numerically that random high-dimensional vectors are nearly orthogonal
Rank toy "documents" by cosine similarity — a hand-built preview of embedding search

Today's ~125 minutes

Spaced-rep warm-up: due flashcards from Week 7	10 min
ELI5 + tech read + 3B1B chapters 1–3, vector-dot visualizer	25 min
Guided: three faces of the dot product + high-dim sim	45 min
Practice: build the similarity toolkit	20 min
Project: similarity lab notebook	15 min
Quiz + flashcards	10 min

Builds on: Day 22 — Big O — cost of comparing everything · Day 4 — Collections & list operations

The analogy

🧸 ELI5⚙ Tech

Picture an arrow on the ground pointing somewhere, and the sun directly overhead one side of it. The dot product of two arrows asks: if I shine light along arrow B, how long is the shadow arrow A casts on it — scaled by both lengths? Arrows pointing the same way cast long shadows on each other (big positive dot product). Perpendicular arrows cast no shadow at all (dot product zero — they share nothing). Arrows pointing opposite ways cast "negative shadow" (negative dot product).

Now the leap that powers modern AI: an arrow does not have to live in 2D. A list of 768 numbers is an arrow in 768-dimensional space, and the shadow trick still works, computed the same way: multiply matching coordinates, add them up. If you place words at arrow-tips so that similar meanings point in similar directions — which is exactly what an embedding model does — then "how similar are these two sentences?" becomes "how aligned are these two arrows?", one multiply-add away. Cosine similarity is the shadow question with lengths divided out: pure direction, ignoring how long the arrows are. Today you build that machinery with your own hands in NumPy; on Day 92 a neural network will build the arrows for you.

Why this matters on the job

The dot product is arguably the single most-executed operation in AI: every neuron (Day 85), every attention score (Day 94), and every vector-database query (Day 115) is dot products at industrial scale. When you ship RAG systems, "the retriever returns junk" debugging starts with cosine similarities between query and chunk embeddings — numbers you must read fluently. And the high-dimensional intuition you build today (near-orthogonality of random vectors) is why embedding spaces can host millions of distinct meanings without collapsing — the geometric fact that makes semantic search work at all.

Watch it happen

Dot product = how much two vectors agree

step 1 / 6

	action	romance	scifi
Ana	5	1	4
Ben	4	0	5

Two vectors describing movie tastes: [action, romance, scifi]. Ana = [5, 1, 4], Ben = [4, 0, 5].

🎯 Predict modespeed1×

Guided practice

guided 1

Arrows, shadows, and the three faces of the dot product

25 min

Run part 1: two 2D vectors, their norms, and the dot product computed BOTH ways — component sum and ‖v‖‖w‖cos θ. Confirm they match to floating-point precision.
Rotate w around the circle in 30° steps (part 2) and watch the dot product trace out alignment: maximal when parallel, zero at 90°, most negative when opposite. Say the shadow story out loud at each step.
Part 3: cosine similarity. Scale v by 100 and confirm the cosine does NOT change (direction is length-blind) while the raw dot product balloons ×100.
Predict, then check: cosine similarity between [1, 0, 1, 0] and [0, 1, 0, 1]? (They share no active components…) Then between [1, 2, 3] and [2, 4, 6]?

🐍 python — editable, runs in your browser

import numpy as np

v = np.array([3.0, 1.0])
w = np.array([2.0, 2.0])

# --- 1. one number, three faces
dot_alg = float(v @ w)                      # sum of componentwise products
cos_theta = dot_alg / (np.linalg.norm(v) * np.linalg.norm(w))
theta = np.degrees(np.arccos(cos_theta))
print(f"v·w = {dot_alg:.3f}   |v|={np.linalg.norm(v):.3f} "
      f"|w|={np.linalg.norm(w):.3f}  angle={theta:.1f} deg")
dot_geo = np.linalg.norm(v) * np.linalg.norm(w) * np.cos(np.radians(theta))
print(f"geometric formula gives {dot_geo:.3f}  (same number)")

# --- 2. sweep w around the circle: alignment as a dial
print("\nangle   v·w     story")
for deg in range(0, 361, 30):
    rad = np.radians(deg)
    w_rot = 2.0 * np.array([np.cos(rad), np.sin(rad)])   # length 2, rotating
    d = float(v @ w_rot)
    story = "aligned" if d > 3 else ("opposed" if d < -3 else
            "~orthogonal" if abs(d) < 1e-9 or abs(d) < 1.2 else "between")
    print(f"{deg:5d}  {d:7.2f}   {story}")

# --- 3. cosine similarity ignores length
def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

print(f"\ncos(v, w)      = {cosine(v, w):.4f}")
print(f"cos(100v, w)   = {cosine(100 * v, w):.4f}   <- unchanged")
print(f"raw dot 100v·w = {float(100 * v @ w):.1f}    <- 100x louder")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

High-dimensional intuition + tiny semantic search

20 min

Part 1 samples 1,000 pairs of random unit vectors in dimensions 3, 32, and 768 and prints the mean and spread of their cosine similarities. Watch the spread shrink like 1/√n — in 768-D, random directions are essentially perpendicular. Write the takeaway in your notes: high-dim space has room for "everything unrelated to everything."
Part 2 is a semantic search engine you can hold in your head: five "documents" with HAND-BUILT 4-dim vectors, where each dimension is an interpretable feature (animal-ness, food-ness, tech-ness, sports-ness). Read the vectors and check they match your intuition about each doc.
Run the query and confirm the ranking makes sense. Change the query vector to "sports tech" (e.g. [0, 0, 0.7, 0.7]) and re-rank.
The punchline to say out loud: Day 92 replaces the hand-built column meanings with 768 learned ones — the search code will not change.

🐍 python — editable, runs in your browser

import numpy as np

rng = np.random.default_rng(50)

def unit(x):
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

# --- 1. random directions barely align in high dimension
for n in (3, 32, 768):
    a = unit(rng.normal(size=(1000, n)))
    b = unit(rng.normal(size=(1000, n)))
    cos = np.sum(a * b, axis=1)             # rowwise dot of unit vectors
    print(f"dim {n:4d}: mean cos {cos.mean():+.3f}  "
          f"spread {cos.std():.3f}  (~1/sqrt(n)={1/np.sqrt(n):.3f})")

# --- 2. hand-built embeddings: [animal, food, tech, sports]
docs = {
    "cats chase mice":        np.array([0.9, 0.1, 0.0, 0.1]),
    "best pizza in town":     np.array([0.0, 0.9, 0.0, 0.0]),
    "new GPU benchmarks":     np.array([0.0, 0.0, 0.9, 0.1]),
    "the match went to penalties": np.array([0.0, 0.1, 0.1, 0.9]),
    "dogs love tennis balls": np.array([0.8, 0.0, 0.0, 0.5]),
}

def search(query_vec, docs):
    q = query_vec / np.linalg.norm(query_vec)
    scored = [(float(q @ (v / np.linalg.norm(v))), text)
              for text, v in docs.items()]
    return sorted(scored, reverse=True)

print('\nquery: "pets" -> [0.9, 0, 0, 0.2]')
for score, text in search(np.array([0.9, 0.0, 0.0, 0.2]), docs):
    print(f"  {score:.3f}  {text}")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Build the similarity toolkit

20 min

Write simtools.py from scratch (no peeking at the guided code): norm(v), cosine(a, b), and top_k(query, matrix, k) where matrix is shape (num_docs, dim) and the function returns the indices and scores of the k most similar rows — computed with ONE matrix-vector product, no Python loop.

Constraints: handle the zero-vector edge case in cosine (return 0.0, do not divide by zero); top_k must use np.argsort (or argpartition) on a single matrix @ query of normalized rows; include three asserts: cosine of a vector with itself is 1.0, with its negative is −1.0, and scaling either argument changes nothing.

Hints (only if stuck): normalize the matrix rows once with keepdims=True; argsort ascending → take the last k reversed.

Ship before you stop

Similarity lab notebook

Create vectors_lab.md + simtools.py in your practice repo. The notebook records four experiments with numbers and one-paragraph readings: (1) the rotating-dot-product sweep table and the shadow story in your own words; (2) the dimension-vs-spread table from the near-orthogonality sim, with the 1/√n observation; (3) your hand-built 4-dim search: the doc vectors, two queries, rankings, and one sentence on WHY each ranking came out as it did; (4) a "gotcha" experiment: two vectors with high cosine similarity but very different norms — explain when that distinction matters (hint: it is why some systems use dot product and others cosine). Close with three sentences titled "what Day 92 will change" (learned dimensions, 768 of them, same math). This file is the seed of your embedding intuition — Day 92 and Day 115 both reference it.

Rubric — check what you completed (0/6)

simtools.py passes its asserts and top_k uses one matmul, zero Python loopsSweep table shows dot product peaking aligned, zero near 90°, negative opposedNear-orthogonality table covers dims 3/32/768 with the 1/√n noteBoth search rankings recorded with a causal explanation eachCosine-vs-dot-product gotcha explained correctlyCommitted with a descriptive message

Common mistakes & misconceptions

Treating vectors as "just lists." The geometry is the point: without the arrow picture, dot products are arbitrary arithmetic and cosine similarity is a magic spell. Always be able to say what the number MEANS.
Confusing dot product with cosine similarity. Dot product scales with both lengths; cosine divides them out. On unit vectors they coincide — which is exactly why pipelines normalize.
Expecting "unrelated" to mean cosine ≈ 0 in real embedding spaces. Real models put unrelated text at 0.1–0.3; read scores comparatively within one model, never as absolute thresholds across models.
Assuming 3D intuition scales. In high dimensions random vectors are near-orthogonal and distances concentrate — some 3D instincts invert. Trust the simulations you ran today.
Looping over rows to compute similarities. One matrix-vector product does all documents at once — this habit (vectorize!) is Day 64's whole lesson and the reason GPUs exist (Day 51).
Forgetting the zero-vector edge case. An all-zeros vector has no direction; cosine with it is undefined. Production code returns 0 or raises — deciding is your job, dividing by zero is not.

Knowledge check

Q1. Two vectors have dot product 0. Geometrically, this means…

Q2. Why do embedding pipelines usually normalize vectors to unit length?

Q3. You sample two random unit vectors in ℝ⁷⁶⁸. Their cosine similarity is most likely…

Go deeper — curated resources

video3Blue1Brown — Essence of Linear Algebra, chapters 1–3 (vectors, span, dot products) ↗35 min bookImmersive Linear Algebra — ch. 2–3 (vectors & the dot product, interactive) ↗25 min docsNumPy — the absolute basics for beginners ↗20 min bookMathematics for Machine Learning — ch. 3 (analytic geometry) ↗25 min

If you have a third hour

The curse (and blessing) of dimensionality — Distances concentrate in high dimensions: the gap between "nearest" and "farthest" shrinks relatively. It complicates naive nearest-neighbor but the near-orthogonality blessing gives models astronomical representational room. Day 79 (PCA) returns here.

Done means

Both guided labs run; dot-product sweep story told at three angles
simtools.py committed, asserts pass, top_k is loop-free
Near-orthogonality table recorded for dims 3/32/768
Lab notebook committed with all four experiments + the Day 92 paragraph
Quiz ≥ 2/3

How this connects

← Back: Day 22's complexity lens applies immediately: scoring n documents against a query is O(n·d) — fine at toy scale, and the reason Day 115's approximate indexes will exist. The vectorized top_k continues the "replace loops with structure" instinct from Day 4.

Forward →: Day 51 stacks many dot products into matrices — a matrix-vector product is just every row shadow-testing the same vector at once, which is also what attention does with queries and keys on Day 94. Day 92 delivers the learned 768-dim vectors, Day 93 trains word vectors, and Day 115 scales today's top_k to millions of documents.

Unlocks: D51 Matrices & Transformations · D52 Rank, Eigenvectors & SVD Intuition · D53 Derivatives & Gradients · D56 Week 8 Checkpoint: Linear Regression by Hand