Attention
- Explain the bottleneck attention solves: fixed vectors cannot carry context-dependent meaning
- Describe queries, keys, and values as a soft, differentiable lookup
- Hand-compute scaled dot-product attention on a 4-token example, exactly
- Implement attention in NumPy and verify it against your hand computation
- Explain why scores are scaled by √d_k and what multiple heads add
| Spaced-rep warm-up: Days 92–93 cards | 10 min |
| ELI5 + tech read; 3B1B attention chapter; attention-heat visualizer | 30 min |
| Guided: hand computation, NumPy verification + heatmaps | 45 min |
| Practice: the jam disambiguation | 20 min |
| Quiz + flashcards + commit attention_lab.py | 15 min |
Builds on: Day 92 — Embeddings & dot-product similarity · Day 93 — word2vec's polysemy ceiling · Day 62 — Softmax & distributions
Watch a good meeting. Before speaking, each person glances around the table and weighs everyone's input: for THIS question, the finance person's comment matters 45%, the two engineers 22% each, the intern's 11%. Their eventual statement is a blend of what they heard, weighted by relevance — and for the NEXT question, the weights change completely. Nobody has a fixed importance; importance is negotiated per question.
Attention runs this meeting between words. Yesterday's problem: "jam" had one frozen vector, torn between traffic and strawberries. Attention thaws it. Every token broadcasts three things: a query — "here's what I'm looking for" — a key — "here's what I offer" — and a value — "here's my actual content." Each token compares its query with everyone's keys (a dot product — Day 92's similarity), converts the match scores into percentages with a softmax, and then takes a weighted blend of everyone's values. In "traffic jam," the token "jam" finds "traffic" highly relevant and blends it in; its output vector now MEANS traffic-jam. Same word, different sentence, different vector. That per-occurrence renegotiation — everyone looks at everyone, every time — is the single mechanism the last decade of AI is built on, and today you compute one by hand, to the third decimal.
Attention is the "T" in GPT and the operation your future stack spends most of its money on: context windows are priced by it, long-document costs explode quadratically because of it (Day 101), and KV caching (Day 155) exists to avoid recomputing it. When you debug a model that ignores the middle of a long prompt or explain to a customer why doubling context more than doubles cost, you are reasoning about today's mechanism. Interviews test it constantly, and the difference between reciting "queries, keys, values" and having COMPUTED one is immediately audible.
Attention weights — every token looks at every earlier token
step 1 / 5| The | cat | sat | down | |
|---|---|---|---|---|
| The | 1.0 | — | — | — |
| cat | .2 | .8 | — | — |
| sat | .1 | .6 | .3 | — |
| down | .05 | .15 | .7 | .1 |
Four tokens: "The cat sat down". Each row asks: while processing ME, how much should I look at each column?the — cells: causal mask. A token can't see the future.
Guided practice
The hand computation — every digit yours
25 min- On paper, reproduce the worked example WITHOUT looking at the tech section: given q4 = [1,1], keys k1..k4 = [1,0],[0,1],[1,1],[0,0], values v1..v4 = [1,0],[0,2],[2,2],[0,0].
- Compute the four dot-product scores. Scale by √2. (Keep 3 decimals.)
- Softmax: exponentiate (e^0.707 ≈ 2.028, e^1.414 ≈ 4.113), sum, divide. You must get [0.221, 0.221, 0.449, 0.109].
- Blend the values: output ≈ [1.12, 1.34]. Check against the tech section only when done.
- Repeat for token 2, whose query is q2 = [0, 2]: scores come out [0, 2, 2, 0], scaled to [0, 1.414, 1.414, 0]. Finish the softmax and blend yourself — you should land on weights ≈ [0.098, 0.402, 0.402, 0.098]. Exercise 2's code will confirm.
- One sentence: why must each row of attention weights sum to 1?
NumPy attention — verify, then heatmap
20 min- Implement
attention(Q, K, V)in NumPy exactly as the formula reads: scores = Q @ K.T / sqrt(d_k); rowwise softmax; weights @ V. - Run it on the full 4-token example (starter) and confirm row 4 of the weights is [0.221, 0.221, 0.449, 0.109] and row 4 of the output is [1.12, 1.34] — your paper, vindicated. Also read off row 2 to check your step-5 answer.
- Print an ASCII heatmap of the weight matrix (starter helper): darker = higher weight. Identify visually which token attends most to which.
- Remove the √d_k scaling and reprint the heatmap: rows sharpen. Now set d_k = 64 with random Q/K scaled up accordingly and compare softmax outputs with and without scaling — saturation appears.
On your own
Make "jam" mean two things
20 minClose the loop on Day 93's polysemy demo — with hand-set vectors, no training. Design a 3-token mini-world: token "jam" plus context token "traffic" OR "strawberry". Give "jam" a neutral value vector, give the two context words value vectors pointing in opposite directions (e.g. [3, 0] vs [0, 3]), and set keys/queries so that jam's query matches whichever context token is present. Run your attention() on the sequence [traffic, jam] and on [strawberry, jam], and show jam's OUTPUT vector differs strongly between the two — the same input embedding, disambiguated by context.
Constraints: use your attention() unchanged; print both output vectors and their cosine similarity (should be low).
Hints: give both context tokens the same key (say [1, 0]) and jam a query aligned with it ([2, 0]) — the mechanism does the rest through the values.
attention_lab.py — the artifact interviews are made of
Commit attention_lab.py: softmax (stable), attention() with a scale toggle, the verified 4-token example asserting the hand-computed row ([0.221, 0.221, 0.449, 0.109] and [1.12, 1.34] within 0.005), the ASCII heatmap, the d_k = 64 saturation demonstration (print max softmax weight with and without scaling), and the jam-disambiguation demo. Add a 10-line README section: the meeting analogy in your own words, then Q/K/V in one sentence each. Day 95 imports attention() into a transformer block and Day 97 reimplements it with trainable weights in PyTorch — this file is the reference they are checked against.
Common mistakes & misconceptions
- Memorizing "Q, K, V" without roles. Query = what I seek; key = what I advertise; value = what I hand over if matched. If you can't assign these in a sentence, re-run the meeting analogy.
- Forgetting softmax normalizes per ROW. Each token distributes exactly 100% of its attention across all tokens; columns need not (and don't) sum to anything.
- Dropping the √d_k "because it barely changes my toy numbers." At d_k = 64+ it is the difference between training and a saturated, gradient-dead softmax.
- Thinking attention weights are THE explanation of model behavior. They show information routing in one layer of many; treat heatmaps as evidence, not verdicts.
- Believing values are redundant with keys. Keys are for MATCHING, values are the CONTENT delivered — the jam demo only works because they differ.
- Missing the quadratic cost: n tokens → n×n score matrix. This single fact drives context pricing, long-context research, and Day 155's KV cache.
Q1. In the worked example, token 4's weights were [0.221, 0.221, 0.449, 0.109]. What produced the 0.449?
Q2. Why divide scores by √d_k?
Q3. Doubling the sequence length from 1,000 to 2,000 tokens does what to the attention score computation?
Go deeper — curated resources
- Cross-attention vs self-attention — In translation-style encoder-decoders, queries come from one sequence and keys/values from another — the same formula, different table. You'll recognize it instantly in the Illustrated Transformer's decoder figures tomorrow.
- Hand computation done unaided and verified to 3 decimals (both rows)
- attention() asserts against the worked example; heatmaps rendered and read
- Saturation demo and jam demo both committed in attention_lab.py
- Quiz ≥ 2/3
← Back: The scores are Day 92's dot-product similarity; the weighting is Day 62's softmax; the saturation danger is Day 86's vanishing-gradient lesson; and the whole mechanism answers the polysemy ceiling Day 93 ended on.
Forward →: Day 95 wraps attention with positions, residuals, and FFNs into the transformer block; Day 97 gives Q/K/V trainable weights in your GPT; Day 101 prices the n² you met today; Day 155's KV cache banks the K and V you just computed.
Unlocks: D95 The Transformer · D97 Tiny GPT Lab I — Build It · D98 Week 14 Checkpoint: Attention, Locked In · D101 Scaling Laws, Capabilities & Limits