Week 14 Checkpoint: Attention, Locked In
- Reproduce the week's core machinery from memory: embeddings, attention arithmetic, the block stack, and BPE
- Explain attention twice — once in pure analogy, once in precise Q/K/V terms — and self-grade both against a rubric
- Retake the week's weak quiz areas and clear the due flashcard deck
- Verify your Day 97 model is ready for training day with a pre-flight checklist
| Blank-page rebuild of the week (closed book) + diff | 30 min |
| Dual-register attention drill with rubric grading | 20 min |
| Pre-flight the Day 97 model for training day | 15 min |
| Flashcard clearance + weak-day requizzes | 20 min |
| Assemble and commit the attention dossier | 25 min |
| Today's quiz | 10 min |
Builds on: Day 92 — Embeddings — meaning as geometry · Day 94 — Attention — Q/K/V · Day 95 — The transformer · Day 97 — Tiny GPT — built and verified
A craftsman ends the week the same way every time: tools cleaned, laid out on the bench, and named out loud — because next Monday's job (training your GPT) depends on grabbing each one without looking. This week you acquired the most valuable tools in the modern AI toolbox: the map of meaning (embeddings), the everyone-looks-at-everyone trick (attention), the assembly line that stacks it (the transformer), the LLM's alphabet (tokens), and yesterday, a working engine built from all four.
Today you close the lid properly. Not by rereading — rereading is watching someone else clean tools — but by the blank-page test: shut every note and reproduce the week from memory. Draw the attention table for four tokens. Write the block wiring. Hand-run BPE's first merge on a tiny corpus. Where your memory serves the answer instantly, that tool is truly yours. Where it hesitates, you have found this week's revision target — better to find it today than mid-interview, or mid-demo when a customer architect asks "so how does attention actually work?"
"Explain attention to me" is the single most common LLM-fundamentals interview prompt in 2026 loops — asked at ELI5 level to test communication, then at tech level to test depth, exactly the two registers you practice today. The pre-flight ritual matters professionally too: FDEs and platform engineers alike learn to verify systems BEFORE the important run, because discovering a broken checkpoint during Monday's training session (or a customer demo) costs credibility that a 10-minute Friday checklist would have saved.
Attention weights — every token looks at every earlier token
step 1 / 5| The | cat | sat | down | |
|---|---|---|---|---|
| The | 1.0 | — | — | — |
| cat | .2 | .8 | — | — |
| sat | .1 | .6 | .3 | — |
| down | .05 | .15 | .7 | .1 |
Four tokens: "The cat sat down". Each row asks: while processing ME, how much should I look at each column?the — cells: causal mask. A token can't see the future.
Guided practice
Blank page: rebuild the week from memory
30 minClose every note, lesson, and file. On paper or a blank editor:
- Attention arithmetic (10 min): for tokens ["the","cat","sat"], write the full computation — Q/K/V from where, score formula, scaling, mask, softmax, weighted sum. Invent small numbers and run one row end to end, Day 94 style.
- Block diagram (5 min): draw the decoder block wiring exactly — where the residuals attach, where the layer norms sit (pre-norm), what the FFN's two layers do to the width.
- BPE first merge (5 min): corpus "low low lower" — what pair merges first and what does the vocab gain?
- Model checklist (5 min): list the 5 components of your Day 97 TinyGPT in forward-pass order with tensor shapes.
- Diff (5 min): open the week's notes and your model.py. Mark every gap or error in red. Each red mark maps to a day — that is your repair list for the flashcard block.
The dual-register drill — explain attention twice
20 min- Set a 90-second timer. Explain attention OUT LOUD in pure ELI5 (no jargon) — the spotlight/meeting-room analogy or your own. Record it or write it verbatim after.
- Set another 90 seconds. Explain it again in precise technical language: Q/K/V, scaled dot-product, mask, softmax, multi-head.
- Grade both against the 4-criteria rubrics in today's tech section. Be harsh — a criterion is only met if a stranger would agree.
- Repair the weakest criterion: re-read ONLY the relevant Day 94 section, then redo that register once. One focused rep beats three vague ones.
- Save both final versions in your notes — they are your Day 105 stakeholder-explainer and Day 179 interview raw material.
Pre-flight your GPT for training day
15 minRun the Day 97 verification harness fresh, as if you did not trust yesterday's self:
python tinygpt/model.py— all three checks (init loss ≈ ln V, leak test, one-batch overfit) print PASS.- Confirm your corpus file loads, and encode→decode round-trips a sample line losslessly (Day 96's invariant).
- Time 100 forward+backward steps on a (16, 64) batch. Write the seconds/step in your notes — Day 99 uses it to size the real run.
- Commit any fixes. A dirty working tree the night before training day is how "which version did I train?" mysteries begin (Day 81's reproducibility rule).
On your own
Flashcard clearance + weak-day requiz
20 min- Clear the entire due deck on the Review page (Days 92–97 cards are due today by design).
- Any card missed twice becomes a written entry: the concept, and the one-line cue that would have retrieved it.
- Retake the quiz on your two weakest days of the week (lowest scores or most red marks from the blank-page diff). The mastery tracker records the retake — latest score counts, so repairs pay off immediately.
Hints: hesitation counts as a miss — the interview clock and the customer meeting do not accept "give me a second, I knew this".
Week 14 closeout: the attention dossier
Assemble week14_dossier.md in your journey repo: your corrected blank-page attention computation (photographed or retyped), the block diagram, both graded explanation transcripts with rubric scores, the pre-flight results table (three PASSes + seconds/step), and a 5-line "what I'd teach someone starting this week" summary. This dossier is deliberately portfolio-shaped: it is evidence you understand transformer internals, written in your own words — exactly what a hiring manager skims for and what Day 179's interview prep will mine.
Common mistakes & misconceptions
- Rereading instead of recalling. Recognition feels like knowledge; only production under a closed book is. The blank page is uncomfortable precisely because it is working.
- Grading your own explanations gently. The rubric exists to find failures cheaply — a criterion "sort of" met today is a stumble in the interview or the customer meeting.
- Skipping the pre-flight because "it passed yesterday". Files drift, corpora move, working trees dirty. Ten minutes today insures two hours on Day 99.
- Treating review days as rest days. Retrieval, explanation, and verification are the highest-leverage two hours of the week — the lessons only become durable here.
- Explaining attention with an analogy that does not map (e.g. "it's like the model thinking hard"). Every analogy element must correspond to a mechanism, or it teaches vibes instead of structure.
Q1. In the attention computation, what plays the role of "what I am looking for" vs "what I contain"?
Q2. Why does the transformer need position embeddings at all?
Q3. BPE builds its vocabulary by repeatedly…
Go deeper — curated resources
- Blank-page rebuild done closed-book, then diffed with every gap repaired
- Both attention explanations scored ≥ 3/4 on their rubrics after repair
- Pre-flight 3/3 PASS with seconds/step recorded
- Due deck cleared and weak-day quizzes retaken
- week14_dossier.md committed
← Back: Everything drilled today was built this week: Day 92's geometry, Day 93's word2vec, Day 94's attention table (redrawn from memory today), Day 95's block stack, Day 96's BPE merge, Day 97's verified build.
Forward →: Day 99 trains the model you pre-flighted tonight. Day 105's phase assessment reuses your dual-register explanation, and Day 179's interview gym pulls the dossier straight into mock-interview answers.