Embeddings — Meaning as Geometry
- Contrast one-hot and dense representations and say what similarity each can express
- Compute cosine similarity between embeddings and rank nearest neighbors in NumPy
- Demonstrate analogy structure (king − man + woman ≈ queen) with vector arithmetic
- Distinguish word embeddings from sentence embeddings and name a use for each
- Explain why domain shift degrades embedding quality and how you would detect it
| Spaced-rep warm-up: Day 50 cosine cards + Week 13 due cards | 10 min |
| ELI5 + tech read; explore the embed-space visualizer | 25 min |
| Guided: cosine + neighbors, analogies + PCA map | 40 min |
| Practice: the 30-line semantic search engine | 20 min |
| Project: assemble and narrate embedding_geometry.py | 15 min |
| Quiz + flashcards | 10 min |
Builds on: Day 50 — Dot products & cosine similarity · Day 79 — PCA for visualization · Day 85 — Layers & learned representations
Imagine a map where cities are placed not by geography but by meaning. "Coffee" sits a short walk from "espresso," a moderate stroll from "breakfast," and a transcontinental flight from "carburetor." Nobody drew this map by hand — it was learned by reading billions of sentences and nudging words that keep similar company (Day 93 shows exactly how) until distance on the map matched relatedness in life.
An embedding is a point on that map: a list of a few hundred coordinates for each word, sentence, or document. The magic is that the map's directions mean something too. The walk from "man" to "woman" is roughly the same arrow as from "king" to "queen" — so arithmetic like king − man + woman lands you near "queen." Compare that with the naive alternative: giving every word its own unrelated ID (one-hot), where every pair of words is exactly equally far apart and "coffee" is as unrelated to "espresso" as to "carburetor." The map is why modern search can find "how do I get my money back" in a document titled "refund policy" — no shared words, close coordinates. Every LLM starts by placing your tokens on such a map; everything after is geometry.
Embeddings are the single load-bearing abstraction of applied AI: retrieval (Day 115's vector database is literally a warehouse of these points), semantic search, deduplication, recommendation, clustering support tickets, and the first layer of every transformer. As an FDE you will pick embedding models for customers, explain why search "understands synonyms," and debug the day it stops working — usually domain shift: a model trained on web prose making a jumbled map of medical or legal jargon. Today's geometry vocabulary — cosine, neighbors, drift — is the working language of half your future projects.
The map of meaning — words become coordinates
step 1 / 6To a computer, words start as arbitrary IDs. Is 4021 similar to 8817? The question is meaningless — IDs have no geometry, so "cat" and "kitten" are perfect strangers.
Guided practice
Build the map by hand — cosine and neighbors
20 min- The starter defines a tiny 8-word embedding table (3-dim, hand-set so the geometry is legible).
- Before running: predict the nearest neighbor of "coffee" and whether "coffee"–"tea" beats "coffee"–"engine".
- Implement cosine_sim(a, b), then most_similar(word, k) that ranks the other 7 by cosine. Confirm predictions.
- Now one-hot the same vocabulary (8-dim identity) and compute all pairwise cosines. Observe: every off-diagonal is exactly 0 — one-hot cannot rank ANY similarity.
- Normalize the dense table once (divide rows by norms), recompute all similarities as sims = E @ E.T, and confirm it matches your pairwise loop.
Analogy arithmetic and a 2D map
20 min- Extend the table with gendered royalty (starter): man, woman, king, queen, prince, princess, with coordinates where dimension 2 encodes "royalty" and dimension 1 encodes gender.
- Compute v = E[king] − E[man] + E[woman] and rank all words by cosine to v (excluding the three inputs). "queen" must win. Then try prince − man + woman.
- Project the full 14-word table to 2D with PCA (Day 79 payoff — sklearn's PCA or your own SVD) and print/plot coordinates. The drink cluster, the car cluster, and the royalty axis should be visibly separated.
- In one sentence: what property of the SPACE (not the individual points) makes the analogy work?
On your own
A semantic search engine in 30 lines
20 minBuild search(query_vec, doc_vecs, k) over a toy corpus: 12 one-line "documents" you write across three topics (e.g. billing, shipping, technical support), each assigned a hand-crafted 4-dim embedding where each topic owns a direction (leave one dimension for "urgency" and give two docs high urgency). Return the top-k docs with scores. Then demonstrate: (1) a billing-flavored query vector retrieves billing docs even though you never string-match; (2) an off-map query (all zeros except the unused corner of the space) returns garbage rankings with LOW top scores — and write one sentence on why a production system should threshold on the score rather than blindly take top-k.
Constraints: normalize everything once; the ranking must be a single matmul plus argsort. Hints: np.argsort(-scores)[:k]; the garbage case is domain shift in miniature.
embedding_geometry.py — the demo you will reuse in customer calls
Commit embedding_geometry.py: the hand-set vocabulary, cosine/most_similar/analogy functions, the one-hot contrast, the PCA map, and the toy semantic-search engine with its domain-shift demonstration — each behind a small function with a docstring, orchestrated by a __main__ that narrates the story in printed sections. Requirement: someone who has never heard the word "embedding" can run this file top to bottom and follow the printed narrative. This is genuinely a demo you can re-give on Day 115 (with a real vector DB) and in FDE conversations; write the prints like you are presenting.
Common mistakes & misconceptions
- Using Euclidean distance on unnormalized embeddings and wondering why long documents cluster together. Magnitude often tracks length/frequency; cosine (or normalize-then-dot) compares direction — meaning.
- Believing one vector per word is enough. "Bank" has one point but two meanings; fixed embeddings average them. Contextual embeddings (Day 95) exist for exactly this.
- Treating analogy arithmetic as a reliable production tool. It is a beautiful diagnostic of space structure, not an API — it degrades off the famous examples.
- Comparing embeddings produced by DIFFERENT models. Each model learns its own coordinate system; cosine across models is meaningless. Re-embed everything when you switch models (a real re-indexing cost on Day 115).
- Assuming an embedding model works on your domain because its benchmark scores are high. Benchmarks are web-prose; your customer's jargon may be off the map. Spot-check neighbors before trusting retrieval.
- Forgetting that nn.Embedding is trained, not given. The map is learned by gradient descent like every other layer — tomorrow you train one yourself.
Q1. Why can't one-hot vectors express that "coffee" and "espresso" are related?
Q2. Your customer's search retrieves nonsense for their drilling-equipment jargon, though it works fine on general questions. Most likely cause?
Q3. king − man + woman ≈ queen works because…
Go deeper — curated resources
- Embed real sentences locally ↗ — pip install sentence-transformers, load all-MiniLM-L6-v2, embed 20 sentences of your own across 3 topics, and rerun today's PCA map on REAL vectors. Watching your hand-built geometry appear in a production model is the payoff.
- most_similar, analogy, and the one-hot contrast all reproduce documented results
- PCA map rendered with three visible clusters
- Search engine demo works, including the domain-shift low-score case
- embedding_geometry.py committed as a runnable narrated demo
- Quiz ≥ 2/3
← Back: Cosine similarity is Day 50's alignment score finally meeting its destiny; the 2D map is Day 79's PCA; and nn.Embedding is just a Day-85 layer whose rows are looked up instead of multiplied.
Forward →: Tomorrow (Day 93) you TRAIN one of these maps with word2vec. Day 94's attention computes query-key dot products — retrieval inside the network. Day 115 industrializes today's search loop into a vector database, and Day 136 turns "are the neighbors right?" into formal retrieval evals.
Unlocks: D93 word2vec Lab · D94 Attention · D96 Tokenization · D98 Week 14 Checkpoint: Attention, Locked In