Day 99 · Watching it learn to spell

Tiny GPT Lab II — Train & Sample

You will be able to

Train the Day-97 character-level transformer on a real corpus with a full training loop
Verify the initial loss against the cross-entropy prediction for a uniform distribution
Track train vs validation loss and generate samples at checkpoints to watch quality evolve
Run controlled experiments on context length and temperature and record their effects

Today's ~115 minutes

Spaced-rep warm-up: Week 14 due cards (attention, transformer, tokenization)	10 min
Read the training recipe + sanity-check theory	15 min
Guided: wire the loop, validate initial loss, train	40 min
Guided: sampling + sample diary	15 min
Practice: context-length and temperature ablations	20 min
Ship the repo folder, quiz + flashcards	15 min

Builds on: Day 97 — Tiny GPT Lab I — building the model · Day 88 — Training loops & DataLoaders · Day 96 — Tokenization · Day 62 — Entropy & cross-entropy

The analogy

🧸 ELI5⚙ Tech

Yesterday you built a tiny robot that can hold a pen. Today you teach it to write. At first it produces pure static — random letters, like a toddler mashing a keyboard. Then, after a few hundred practice rounds, something eerie happens: spaces start appearing at word-like intervals. Then real short words show up: "the", "and", "of". Then longer words, then sentence-shaped things with capital letters and periods. It is learning to spell, then to imitate style, purely by playing one game millions of times: "given these characters, guess the next one."

Nothing about the robot changed structurally — it is the same machine you assembled on Day 97. What changed is millions of tiny weight adjustments, each one nudged by the gradient of how surprised the model was by the true next character. Watching gibberish become language over twenty minutes of training is the single best intuition-builder for what happened, at incomprehensible scale, inside every large language model you will ever call through an API.

Why this matters on the job

Every LLM you will deploy, prompt, or debug for the rest of this program was trained exactly like this — same loss, same loop, only scaled up a billionfold. Having personally watched loss fall and samples improve gives you conviction that no blog post can: you will explain "it just predicts the next token" to customers with the authority of someone who trained one. It also cements the practical skills — checkpointing, loss curves, sampling — that reappear on Day 128 when you fine-tune a real model with LoRA.

Guided practice

guided 1

Wire the loop and validate the starting loss

25 min

Download a plain-text corpus (Tiny Shakespeare from the nanoGPT repo works well) as input.txt.
Paste the starter below into train.py next to your Day-97 model.py. Adjust the TinyGPT constructor call to your own signature.
Before the loop runs, print the loss of the untrained model on one batch. Compute ln(vocab_size) with math.log and confirm they match to within ~0.1. Write one sentence in your log explaining WHY (Day 62).
Train 2,000 steps on CPU (or 5,000 on GPU). Record train/val loss every 250 steps in a simple list you print at the end.
Confirm val loss tracks train loss downward. If train falls and val rises, name the phenomenon (Day 89) and note at what step it started.

🐍 python — editable, runs in your browser

import math
import torch
from model import TinyGPT  # your Day-97 model

with open('input.txt', 'r', encoding='utf-8') as f:
    text = f.read()

chars = sorted(set(text))
vocab_size = len(chars)
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: ''.join(itos[i] for i in ids)

data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9 * len(data))
train_data, val_data = data[:n], data[n:]

block_size, batch_size = 128, 64
device = 'cuda' if torch.cuda.is_available() else 'cpu'

def get_batch(split):
    d = train_data if split == 'train' else val_data
    ix = torch.randint(len(d) - block_size - 1, (batch_size,))
    x = torch.stack([d[i:i + block_size] for i in ix])
    y = torch.stack([d[i + 1:i + block_size + 1] for i in ix])
    return x.to(device), y.to(device)

model = TinyGPT(vocab_size=vocab_size, block_size=block_size).to(device)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)

print('expected initial loss ~', round(math.log(vocab_size), 3))

@torch.no_grad()
def eval_loss(split, iters=50):
    model.eval()
    losses = [model(*get_batch(split))[1].item() for _ in range(iters)]
    model.train()
    return sum(losses) / len(losses)

for step in range(2000):
    xb, yb = get_batch('train')
    logits, loss = model(xb, yb)
    opt.zero_grad(set_to_none=True)
    loss.backward()
    opt.step()
    if step % 250 == 0:
        print(step, 'train', round(eval_loss('train'), 3),
              'val', round(eval_loss('val'), 3))

torch.save({'model': model.state_dict(), 'chars': chars},
           'tinygpt_ckpt.pt')

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Watch it learn to spell

15 min

Add the generate function below and call it every 500 steps during training, printing 300 characters from a newline seed.
Keep a "sample diary": paste the step number, current val loss, and the sample into a markdown file.
Annotate each entry with what changed: when did spaces appear? First real word? First correctly matched punctuation pair?
After training, write three sentences connecting loss values to sample quality — this becomes raw material for your Day-105 explainer.

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Two ablations, one table

20 min

Run two controlled experiments and record results in a table (setting, final val loss, one-line sample verdict).

(1) Context length: retrain with block_size 32 vs 128. Which produces more coherent long-range structure (character names, matched quotes)? Why would a model that can only see 32 characters back struggle with a quote opened 80 characters ago?

(2) Temperature: from your best checkpoint, sample at temperature 0.3, 0.8, and 1.5 with the same seed. Describe each in one line.

Hints: keep every other hyperparameter fixed — one variable per experiment (Day 81 discipline). Low temperature repeats safe patterns; high temperature increases entropy of each choice (Day 62).

Today's build

Ship the training run

Turn today's work into a small, reproducible repo folder: train.py, generate.py (loads the checkpoint and takes a --temperature flag via argparse, Day 16 payoff), your sample diary, and a README with the loss table, the ln(vocab_size) sanity-check explanation, and both ablation results. Commit the checkpoint file reference (not the weights if large) and push. This artifact is your proof-of-understanding for Phase 5 — the Day-105 assessment and the Day-128 fine-tuning lab both refer back to it.

Rubric — check what you completed (0/6)

Training runs end-to-end and final val loss is at least 40% below the initial ln(vocab_size) baselineInitial-loss sanity check documented with the correct entropy explanationSample diary shows at least 4 checkpoints with annotations tying loss to sample qualitygenerate.py loads the saved checkpoint standalone and honors a --temperature flagAblation table covers both context length and temperature with one-line conclusionsPushed to GitHub with a README a peer could reproduce from

Common mistakes & misconceptions

Skipping the initial-loss check. ln(vocab_size) at step 0 is your free wiring test — a wrong starting loss means broken shapes or targets, and finding that after an hour of training hurts.
Evaluating with the model in train mode. Dropout stays active and val loss reads noisy-high; wrap evaluation in model.eval() plus torch.no_grad() and switch back.
Forgetting to crop the context to block_size during generation. Positional embeddings only exist for block_size positions — feeding a longer sequence crashes or silently degrades.
Judging the model only by loss. Two checkpoints with similar loss can sample very differently; always look at generations, the habit that becomes "look at your outputs" in LLM evals (Day 134).
Comparing temperature samples from different seeds or checkpoints. Change one variable at a time or the comparison is meaningless.
Training the tokenizer split before splitting train/val on TEXT order but sampling windows across the boundary — keep val windows strictly out of the training slice.

Knowledge check

Q1. Your untrained 65-character model shows an initial loss of about 4.17. Why is that expected?

Q2. During generation you must slice the input to the last block_size tokens because…

Q3. Train loss keeps falling but validation loss starts rising at step 3,000. The model is…

Go deeper — curated resources

courseKarpathy — Zero to Hero ("Let's build GPT" lecture) ↗60 min (skim the training half)reponanoGPT — reference training loop & Tiny Shakespeare data ↗20 min bookDive into Deep Learning — language model training chapters ↗25 min docsPyTorch Tutorials — checkpoint save/load ↗15 min

If you have a third hour

Learning-rate warmup and cosine decay — Real LLM runs warm the LR up over the first steps then decay it. Try adding torch.optim.lr_scheduler.CosineAnnealingLR and compare curves — this reappears in every training-config file you will read.

Done means

Initial loss matched ln(vocab_size) and the explanation is written down
Final val loss ≥ 40% below baseline with the loss table recorded
Sample diary shows the gibberish → words → style progression
generate.py works standalone from the checkpoint; repo pushed
Quiz ≥ 2/3

How this connects

← Back: Day 97 built the machine; today Day 88's canonical training loop and Day 62's cross-entropy turned it into a language model. The temperature mechanics you used come straight from the softmax you met on Day 94.

Forward →: Day 100 scales this exact recipe to trillion-token pretraining. Day 102 dissects the sampling knobs you just touched. On Day 128 you will run this loop again — but fine-tuning a real open model with LoRA instead of training from scratch.

Unlocks: D100 How LLMs Are Trained · D101 Scaling Laws, Capabilities & Limits · D102 Decoding & Sampling · D104 Context Windows & Hallucination Deep-Dive