The Transformer
- Draw a transformer block from memory: attention, residuals, layer norm, FFN
- Explain why position must be injected and how positional encodings do it
- Explain causal masking and why decoder-only models train on every position at once
- Distinguish encoder, decoder, and decoder-only architectures and name a model for each
- Estimate a transformer's parameter count from its config (d_model, layers, vocab)
| Spaced-rep warm-up: Day 94 attention cards | 10 min |
| Guided 1: Illustrated Transformer active read-along with redrawing | 25 min |
| Tech read + transformer-stack visualizer | 15 min |
| Guided 2: causal masking + shape trace + napkin math | 25 min |
| Practice: model-card estimates | 20 min |
| Project commit + quiz + flashcards | 20 min |
Builds on: Day 94 — Scaled dot-product attention · Day 89 — Normalization & training stability · Day 85 — MLPs / feed-forward layers
Yesterday you built one meeting room where every word listens to every word. A transformer is the whole office building: the same meeting-then-deskwork floor plan, stacked twelve, forty, ninety-six floors high. On each floor, tokens first hold the attention meeting (gather context from colleagues), then go to their desks for private processing — a small two-layer MLP where each token digests what it heard, alone. Meeting, deskwork, next floor.
Two pieces of office infrastructure make a skyscraper of these floors trainable. First, every meeting and every desk has a bypass corridor: a token's original vector flows AROUND each step and gets the step's output ADDED to it. If a floor has nothing useful to add, the corridor lets information pass unharmed — and, for training, it gives Day 86's blame a clean highway down ninety-six floors instead of dying in the stairwells. Second, at every doorway there's a recalibration station (layer norm) keeping each token's vector in a healthy range, floor after floor.
One oddity to fix at the entrance: the attention meeting is a round table — it has no idea who sits where. "Dog bites man" and "man bites dog" would look identical. So each token's entry badge encodes its seat number: positional information, added to the embedding before floor one.
This architecture IS the product you will spend your career deploying: GPT, Claude, Llama, the embedding models of Day 115, the rerankers of Day 117 — all this floor plan with different configs. Reading a model card ("32 layers, d_model 4096, 32 heads") should conjure a concrete machine and an approximate parameter count in your head. Causal masking — today's subtlest idea — is why LLM training is so efficient and why generation is one-token-at-a-time, which is TTFT and tokens/sec economics when you budget latency for a customer on Day 155.
The assembly line of attention — one token's trip through GPT
step 1 / 6The word "sat" (as token ids) enters the line. Six stations stand between it and a prediction of the next word.
Guided practice
Illustrated Transformer read-along, actively
25 min- Open The Illustrated Transformer and read from the top through the encoder section, slowly.
- As you read, redraw each major figure BY HAND on paper: the QKV projections, the multi-head split, the residual+norm wrapper, the FFN. Redrawing is the active ingredient — no screenshots.
- When you reach the decoder section, write in the margin where the causal mask enters, and which attention layer of the decoder is cross-attention (queries from the decoder, keys/values from the encoder — Day 94's deep-dive).
- Close the article. Draw one complete decoder-only block from memory, labeling every arrow. Check and mark gaps in red.
- File the drawing — Day 98 asks you to redo it cold.
Causal masking in NumPy + trace one token
25 min- Import attention() from your attention_lab.py and add a
causal=Trueoption: build a mask where score[i][j] = -1e9 for j > i, added before softmax. - Run it on Day 94's 4-token example. Verify: row 1 attends 100% to itself; row 4 is UNCHANGED from yesterday ([0.221, 0.221, 0.449, 0.109]) because token 4 could already see everyone. The weight matrix must be lower-triangular.
- Print the ASCII heatmap and confirm the upper triangle is blank.
- Now trace one token through one block on paper, shapes only, d_model = 8, seq = 4: x (4×8) → +position (4×8) → LN → attention (4×8) → +x → LN → FFN 8→32→8 → +. Write every shape at every arrow; nothing changes shape end to end — that invariant is what lets blocks stack.
- Parameter count for this toy block: attention 4·8² = 256, FFN 8·32·2 = 512 (ignore biases) — confirm 12·d² = 768. Then do GPT-2 small on the napkin: 12·12·768² + 50257·768 ≈ 124M.
On your own
Model-card napkin math
20 minUsing only params ≈ 12·L·d² + vocab·d, estimate total parameters for: (a) L=24, d=1024, vocab=50k; (b) L=32, d=4096, vocab=128k; (c) your Day-97 tiny GPT: L=4, d=128, vocab=65. Then answer: (1) for model (b), what fraction of parameters is embeddings vs blocks? (2) if you double d at fixed L, what happens to block parameters? (3) which grows a model faster, doubling L or doubling d, and why?
Constraints: napkin only — no code until you have three written estimates; then verify with a 5-line script.
Hints: (a) ≈ 353M; blocks scale as d² so doubling d quadruples them while doubling L merely doubles them.
The annotated block diagram + masked attention
Commit two artifacts: (1) causal_attention.py — the masked attention with assertions that row 1 is one-hot and row 4 matches Day 94, plus the lower-triangular heatmap; (2) transformer_notes.md — your from-memory block diagram (photo or ASCII art), the per-arrow shape trace at d_model = 8, the napkin parameter math for GPT-2 small and the three practice configs, and a five-line "why residuals, why layer norm, why mask" section in your own words. These notes are the spec you will build against on Day 97 — every component you can't explain today becomes a bug you can't find tomorrow.
Common mistakes & misconceptions
- Thinking attention does the "thinking." Attention routes information BETWEEN tokens; the FFN (two-thirds of the parameters) processes it per token. Both halves matter.
- Forgetting position entirely — attention alone cannot distinguish "dog bites man" from "man bites dog". If a from-scratch transformer outputs order-independent nonsense, the missing position embedding is the first suspect.
- Confusing layer norm with batch norm. LN is per-token (no batch statistics, no train/eval difference); BN's Day-89 caveats do not apply.
- Placing the causal mask after softmax. The −∞ must enter BEFORE normalization so masked positions get exactly zero weight and rows still sum to 1.
- Believing generation-time causality is a special mode. The mask is there during training too — that is what makes all n positions honest training examples in one pass.
- Reading "96 layers" as 96 different designs. It is one block stamped 96 times — which is why the residual stream must keep its shape, and why your shape trace matters.
Q1. Why do transformers need positional encodings?
Q2. The causal mask sets scores to −∞ for j > i BEFORE softmax so that…
Q3. Roughly how many parameters does a decoder-only model with L=12, d=768, vocab=50k have?
Go deeper — curated resources
- Learned vs sinusoidal vs rotary positions — GPT-2 learns a position table (hard context cap); the 2017 paper used fixed sinusoids; modern models (Llama-family) use RoPE, rotating Q/K by position angle. Worth a search once Day 97's build makes position embeddings concrete.
- Block diagram reproduced from memory with gaps marked and fixed
- Causal attention verified (one-hot row 1, unchanged row 4, lower-triangular map)
- GPT-2 napkin math within 10% and practice configs verified by script
- transformer_notes.md committed as the Day-97 build spec
- Quiz ≥ 2/3
← Back: The block wraps Day 94's attention with Day 85's MLP as the FFN; residuals are Day 86's gradient-highway lesson built into architecture; layer norm is Day 89's normalization idea, per token.
Forward →: Day 96 decides what the tokens ARE. Day 97 turns transformer_notes.md into working code, Day 100 explains how these stacks are trained at scale, Day 101 prices the n² and the context cap you met today, and Day 128's LoRA surgically edits exactly these weight matrices.
Unlocks: D96 Tokenization · D97 Tiny GPT Lab I — Build It · D98 Week 14 Checkpoint: Attention, Locked In · D100 How LLMs Are Trained