word2vec Lab
- State the distributional hypothesis and explain how it turns raw text into a training signal
- Describe skip-gram training: predict context words from a center word
- Explain negative sampling and why it replaces a full-vocabulary softmax
- Train a tiny word2vec from scratch in NumPy and verify that neighbors become sensible
- Name the limitation (one vector per word) that leads to contextual embeddings
| Spaced-rep warm-up: Day 92 geometry cards | 10 min |
| ELI5 + tech read; The Illustrated Word2vec | 30 min |
| Guided: corpus + pairs, SGNS training with before/after | 40 min |
| Practice: the three knob experiments | 20 min |
| Project: polysemy demo + commit | 10 min |
| Quiz + flashcards | 10 min |
Builds on: Day 92 — Embedding geometry & cosine · Day 86 — Gradients & update rules · Day 72 — Sigmoid & logistic loss
"You shall know a word by the company it keeps," wrote the linguist J.R. Firth in 1957 — and half a century later that one sentence trained the first great meaning-maps. You have never tasted "quvra," but if you read "she poured a glass of chilled quvra," "quvra pairs well with fish," and "this quvra is a bit dry," you know it's a wine-like drink. Not from a definition — from its neighbors.
word2vec (Mikolov et al., 2013) mechanizes this. Slide a window over billions of sentences. For each center word, play a guessing game: which words appear near me? Every word starts at a random spot on the map (Day 92's map, before it means anything). Each time two words co-occur, nudge their vectors a little closer; each time you sample a random word that did NOT co-occur, nudge it a little away. That second trick — compare against a few random "impostors" instead of the whole dictionary — is negative sampling, and it's why the whole thing is cheap enough to run on a laptop. Repeat a few million times and "coffee" has drifted next to "tea" without anyone ever defining either. Today you build the entire game in NumPy and watch a random cloud organize itself into meaning.
word2vec is the conceptual ancestor of everything you will deploy: modern sentence embedders and LLM token embeddings are descendants of this exact idea, and "predict the neighbors" is one step from "predict the next token" — Day 97's GPT objective. Practically, negative sampling (turning an intractable softmax into a few binary classifications) is a trick you will re-meet in recommendation systems and contrastive learning. And when a customer asks "how does the machine know synonyms?", the honest, demo-able answer is today's lab — you can train one in front of them in two minutes.
Guided practice
Generate a corpus and build the pair pipeline
15 min- Real corpora need minutes of training; we engineer a tiny one with strong co-occurrence structure so results appear in seconds. The starter generates 400 template sentences across two worlds: beverages and vehicles, sharing function words.
- Build the vocabulary (word ↔ index) and generate skip-gram pairs with window ±2. Print the 5 most common pairs — they should look sensible ("drink" with beverage nouns, etc.).
- Build the negative-sampling distribution: unigram counts to the 0.75 power, normalized. Print the probability of the most and least frequent word under it, and under the raw unigram — see how 0.75 flattens the gap.
- Sanity: count total pairs (should be several thousand).
Train SGNS and watch meaning appear
25 min- Initialize W_in and W_out as (V, 16) with small random values. BEFORE training, print the 3 nearest neighbors of "coffee" using Day 92's cosine — they will be arbitrary. Save this printout.
- Type the training loop from the starter: 5 epochs over shuffled pairs, K=5 negatives per pair, lr 0.05. Note the gradient lines — they are Day 72's logistic gradient, nothing more.
- Every epoch, print the running mean loss; it must fall.
- After training, reprint the neighbors of "coffee", "engine", and "drives". Beverages should cluster, vehicle words should cluster, and the arbitrary "before" printout is your evidence of learning.
- Optional: PCA the 16-dim vectors to 2D (Day 92's code) and admire the two islands.
On your own
Knob experiments
20 minRun three controlled experiments on your SGNS lab (Day 90 discipline — one change, same seed, note the effect on the neighbors of "coffee" and the final loss):
- K = 0 negatives (delete the negative loop). What happens to the space and why? (Think: what stops ALL vectors from collapsing onto each other when there is only attraction?)
- Window ±1 vs ±4 on a corpus where you add 100 mixed sentences ("the car is near the coffee shop"). Which window keeps the clusters cleaner?
- dim = 2 vs dim = 64. Which under- and which over-fits this tiny vocabulary?
Deliverable: 6 lines in your lab notes — experiment, observation, one-sentence explanation each.
Hint for (1): negative sampling provides the repulsive force; without it the trivial solution "make every dot product huge" scores perfectly.
word2vec_lab.py + the limitation demo
Commit word2vec_lab.py: corpus generation, pair pipeline, SGNS training, neighbors(), before/after evidence, and your three knob experiments as toggleable functions. Then add the closing demo, polysemy_demo(): extend the corpus so the word "jam" appears in both worlds ("traffic jam on the road", "strawberry jam with tea"), retrain, and print jam's neighbors — one confused vector stretched between two meanings. End with a printed sentence: "One vector per word cannot represent context-dependent meaning; tomorrow's mechanism fixes this." That printout is your bridge to attention.
Common mistakes & misconceptions
- Thinking word2vec understands language. It compresses co-occurrence statistics into geometry — powerful, but "knows the company words keep," nothing deeper.
- Skipping negatives and wondering why every word becomes every word's neighbor. Attraction-only training collapses the space; negative sampling is the repulsion that keeps it structured.
- Using the full-softmax formulation "to be correct." Its O(V) per-step cost is exactly the problem SGNS was invented to remove; know both, implement one.
- Forgetting there are TWO tables. Center (W_in) and context (W_out) vectors play different roles; you keep W_in (or average them) at the end.
- Expecting textbook analogies from a toy corpus. Analogy structure needs corpus scale and diversity; your lab shows clustering, which is the honest claim at this size.
- Concluding fixed embeddings are obsolete trivia. Their limitation (polysemy) is the WHY of contextual models — interviewers love this exact narrative arc.
Q1. Negative sampling exists primarily to…
Q2. In skip-gram, the model is trained to…
Q3. The word "jam" ends up midway between the traffic cluster and the food cluster. This illustrates…
Go deeper — curated resources
- Train on real text with gensim (local) — pip install gensim; Word2Vec(sentences from any public-domain book, vector_size=100, window=5, negative=10). Real analogies start working around a few million tokens — compare against your toy lab to feel what scale buys.
- SGNS trains with falling loss; before/after neighbors show emerged clusters
- Knob experiments logged (collapse-without-negatives observed and explained)
- polysemy_demo committed with the bridge-to-attention printout
- Quiz ≥ 2/3
← Back: The map you hand-crafted on Day 92 is now LEARNED; the update rule is Day 72's logistic gradient applied with Day 86's mechanics; the one-change experiment discipline is Day 90's lab habit.
Forward →: Tomorrow attention computes context-dependent mixing — the cure for today's polysemy ceiling. Day 96's tokenizers decide what units get embedded, Day 97's GPT trains its embedding table jointly with everything else, and Day 115 retrieves with the industrial descendants of today's vectors.
Unlocks: D94 Attention