AI ENGINEER / 180
Day 97 · Your own GPT, pocket-sized

Tiny GPT Lab I — Build It

You will be able to
  • Assemble a working char-level GPT from parts you already understand: embeddings, attention, FFN blocks, and a head
  • Verify every tensor shape at every stage and explain what each dimension means
  • Apply the causal mask correctly and prove future tokens cannot leak into predictions
  • Overfit a single batch as a sanity test and explain why that proves the wiring is right
Today's ~120 minutes
Spaced-rep warm-up: due cards from Days 92–9610 min
ELI5 + tech read; trace the build order on the visualizer15 min
Guided: embeddings + attention head + block (with leak test)45 min
Guided: assemble TinyGPT + overfit one batch25 min
Practice: break it three ways15 min
Quiz + commit the verified model10 min

Builds on: Day 94 — Attention — Q/K/V and the weight table · Day 95 — The transformer block stack · Day 96 — Tokenization — chars to ids · Day 88 — Training loops & data

The analogy

You have spent two weeks touring an engine factory: the day you saw pistons (attention), the day you saw the fuel system (embeddings), the day you saw the assembly line layout (the block stack). Today you stop touring and build the engine — a miniature one, on your own workbench, from the exact parts you studied. It will be tiny: a few thousand parameters where GPT-4-class models have trillions of times more. That is the point. A model airplane teaches you more about flight than a photo of a 747, because YOU have to make every part fit.

The build order matters, and it is the same trick as testing each Lego section before snapping them together: token embedding first (does a character id become a vector of the right size?), then position embedding, then ONE attention block (do the shapes survive the trip through?), then the stack, then the head that turns the final vectors back into "which character comes next?" scores. After each part, you print the shapes and check them against what you predicted. By tonight you will have a real GPT — untrained, babbling static — but structurally identical to the ones running the world. Tomorrow's checkpoint locks the week in; the day after, you train it and watch it learn to spell.

Why this matters on the job

"Have you actually built a transformer?" separates candidates in every serious AI-engineering loop — not because jobs require writing attention from scratch, but because debugging production LLM systems constantly requires knowing what is inside: why context length costs memory quadratically (the T×T attention matrix you will allocate today), why generation slows as sequences grow, what a logit actually is when you set logit bias in an API call. FDEs get asked "how does this actually work?" by skeptical customer architects; the engineer who has built one answers with earned confidence instead of recited analogy.

Watch it happen

The assembly line of attention — one token's trip through GPT

step 1 / 6
Token idsDay 96→Embed+ position→Attentionlook around→FFNthink alone→× N layersrepeat→Headlogits
✉ "The cat sat" → [464, 2415, 3332]

The word "sat" (as token ids) enters the line. Six stations stand between it and a prediction of the next word.

Guided practice

guided 1

Embeddings in, shapes verified

15 min

Work in tinygpt/model.py (PyTorch, local — torch does not run in the browser interpreter). Karpathy's "Let's build GPT" video is the companion for the whole lab; build alongside it, but type every line yourself.

  1. Load a small corpus (~1 MB of any public-domain text you like). Build the char vocab and encode/decode maps exactly as on Day 96.
  2. Create tok_emb = nn.Embedding(vocab_size, n_embd) and pos_emb = nn.Embedding(block_size, n_embd).
  3. Make one batch (B=4, T=8) of ids. BEFORE running: write down the shape after each of — token lookup, position lookup, their sum. Then print and check.
  4. The sum works via broadcasting: (B,T,C) + (T,C). Say out loud what is being stretched — this is Day 64's broadcasting rule earning its keep.
🐍 python — editable, runs in your browser
Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)
guided 2

One causal attention head, then multi-head, then the block

30 min
  1. Implement a single Head: linear maps to Q, K, V (each C → head_size); scores = Q @ K.T / sqrt(head_size); apply the causal mask (masked_fill on a lower-triangular buffer); softmax; weights @ V. Print the (T,T) weight matrix for one head — row t must be zero after column t. Compare with the numbers you hand-computed on Day 94.
  2. MultiHead: run n_head heads, concat outputs (back to C), one projection layer.
  3. FeedForward: Linear(C, 4C) → ReLU (or GELU) → Linear(4C, C).
  4. Block: pre-norm residual wiring — x = x + sa(ln1(x)) then x = x + ff(ln2(x)). Push your (B,T,C) tensor through one block: same shape out, different values.
  5. Leak test: run the block, change the LAST character of the input, rerun — outputs at positions before the change must be identical. If not, your mask is wrong. This is the test that matters.
🐍 python — editable, runs in your browser
Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)
guided 3

Assemble the full model and overfit one batch

25 min
  1. TinyGPT: embeddings → n_layer Blocks → final LayerNorm → lm_head Linear(C, vocab_size). Forward returns logits (B,T,vocab) and, given targets, cross-entropy loss (reshape to (B*T, vocab) vs (B*T,)).
  2. Instantiate and print the parameter count (sum of p.numel()). Predict it first from the config — being able to budget parameters is a real skill (Day 128's LoRA arithmetic builds on it).
  3. Check initial loss on one batch: should be ≈ ln(vocab_size) ≈ 4.17. Explain why before you run.
  4. Overfit: AdamW, lr=1e-3, the SAME batch, 300 steps. Loss must fall below 0.1. Print a sample generation from the overfit model — it should parrot fragments of that one batch back at you. That parroting is your proof of life.
  5. Commit: tinygpt/model.py with the leak test and the overfit script under if __name__ == '__main__':.

On your own

Break it three ways, predict each failure first

15 min

Sabotage your working model one change at a time, predicting the symptom BEFORE each run, then restore it:

  1. Remove the causal mask entirely. Prediction: what happens to the loss curve on the overfit test, and why is a very low loss here BAD news?
  2. Remove the position embedding. Will the one-batch overfit still succeed? (It can memorize — but what changes in generated text structure?)
  3. Remove the residual connections (x = sa(ln1(x))). Watch the loss. With only 2 layers the damage is mild — state why Day 95 says it becomes fatal at 12+.

Hints: (1) masks prevent an answer-key leak; (2) memorization vs order-awareness are different capabilities; (3) residuals are the gradient highway — depth is what collapses without them.

Today's build

tinygpt/model.py — a verified, untrained GPT

Ship the model file that Day 99 will train. It must contain: the config block at top (vocab_size, n_embd, n_head, n_layer, block_size); Head, MultiHead, FeedForward, Block, and TinyGPT classes; a generate(idx, max_new_tokens) method (loop: crop context to block_size, forward, take last position's logits, softmax, sample, append); and a __main__ harness that runs the three verifications — expected initial loss ≈ ln(vocab), the future-leak test, and the 300-step single-batch overfit — printing PASS/FAIL for each. Commit with the parameter count in the commit message.

Rubric — check what you completed (0/6)

Common mistakes & misconceptions

  • Forgetting the 1/sqrt(head_size) scaling. Softmax saturates on large dot products, gradients vanish, and the model trains poorly — the symptom is subtle, which is why you add it now while everything else is verified.
  • Masking AFTER softmax instead of before. Weights no longer sum to 1 over allowed positions; the leak test may even pass while the math is silently wrong. Mask with −∞ scores, then softmax.
  • Reshaping logits wrong for cross-entropy (mixing up (B*T, vocab) vs (B, vocab, T)). PyTorch's F.cross_entropy expects classes in dim 1 — reshape explicitly and check loss ≈ ln(vocab) at init to catch it.
  • Skipping the overfit test and going straight to full training. Then when loss plateaus you cannot tell wiring bugs from bad hyperparameters — Day 99 becomes archaeology instead of science.
  • Treating tiny scale as "not real". Same equations, same failure modes, same debugging instincts as the frontier models — only the constants differ.
Knowledge check

Q1. At initialization your char-GPT (vocab 65) shows loss 2.1 on the first batch. What does this most likely mean?

Q2. The causal mask sets attention scores to −∞ above the diagonal BEFORE softmax because…

Q3. Your model overfits a single batch to loss 0.05. What have you proven?

Go deeper — curated resources

videoKarpathy — "Let's build GPT: from scratch, in code" (Zero to Hero) ↗1 h of the 2 h video todayreponanoGPT — the grown-up version of what you built ↗15 min skimarticleThe Illustrated Transformer — keep it open as your map ↗referencepaperAttention Is All You Need — now you can read §3 for real ↗20 min
If you have a third hour
  • Read nanoGPT's model.py against yours ↗ — Same skeleton, plus dropout, weight tying, and init tricks. Diffing a professional implementation against your own is a senior-engineer habit worth starting now.
Done means
  • All three __main__ verifications print PASS (init loss, leak test, overfit)
  • You predicted the parameter count within 20% before printing it
  • The three sabotage experiments ran with predictions written BEFORE each run
  • Quiz ≥ 2/3 and model.py committed
How this connects

← Back: This is assembly day: Day 92's embeddings, Day 94's attention (your hand-computed weights reappear in the printed matrix), Day 95's block stack, Day 96's tokenizer, and Day 88's training loop all snap together. The overfit test is Day 89's diagnostics used as a tool.

Forward →: Day 98 locks the week in before Day 99 trains this exact model and you watch gibberish become spelling. The parameter-budget instinct returns in Day 128 (LoRA's low-rank arithmetic), and the KV/attention-cost intuition powers Day 155's serving-latency work.

Unlocks: D98 Week 14 Checkpoint: Attention, Locked In · D99 Tiny GPT Lab II — Train & Sample