Day 102 · The dice behind the words

Decoding & Sampling

You will be able to

Trace the full path from logits through temperature-scaled softmax to a sampled token
Explain greedy, temperature, top-k, and top-p (nucleus) decoding and when each fits
Demonstrate each strategy on your tiny GPT and document the qualitative differences
Explain repetition loops and how penalties and sampling mitigate them
State why LLM output is non-deterministic in practice and what seeds can and cannot fix

Today's ~120 minutes

Spaced-rep warm-up: due cards (scaling, pipeline, entropy)	10 min
ELI5 + tech read; sampling-dist visual	20 min
Guided: decoder zoo on your tiny GPT	25 min
Guided: one-decision temperature/entropy table	15 min
Practice: repetition-loop clinic	20 min
Project: cheat sheet + quiz + flashcards	30 min

Builds on: Day 99 — Tiny GPT training & generation · Day 62 — Entropy & temperature preview · Day 94 — Softmax in attention

The analogy

🧸 ELI5⚙ Tech

The model never chooses a word. Ever. At each step it hands you a scored list of every token in its vocabulary — "the: very likely, cat: likely, xylophone: nearly impossible" — and then walks away. Someone else has to roll the dice. That someone is the decoding strategy, and it is code you can read in twenty lines.

Think of a piano player improvising. Greedy decoding always plays the single most obvious next note: safe, but after a while it loops the same riff forever. Temperature is how adventurous the player feels: cold (0.2) sticks to the obvious notes; hot (1.5) reaches for weird ones, sometimes brilliant, often wrong. Top-k says "only consider the 50 most plausible notes, then roll among them." Top-p is smarter: "consider however many notes it takes to cover 90% of the plausibility — sometimes 3 notes, sometimes 300," adapting to how certain the moment is.

Same model, same weights, wildly different music depending on the dice. When someone says "the model is being creative" or "the model keeps repeating itself," they are usually describing the dice, not the brain.

Why this matters on the job

Sampling settings are the cheapest lever you will ever pull: no retraining, no prompt change, and they can make or break a feature. Extraction pipelines want near-deterministic output; brainstorming tools want diversity; a support bot repeating "I understand your concern. I understand your concern." is a decoding bug you must diagnose on sight. In production you will also field "why did the same prompt give different answers?" from customers — today gives you the precise, honest answer about randomness, seeds, and floating-point non-determinism.

Watch it happen

The dice behind the words — temperature and top-p reshape the roll

step 1 / 6

The model's head outputs a score per token; softmax turns scores into probabilities. Next token after "The cat sat on the": here are the top candidates.mat 46% · floor 22% · couch 14% · chair 8% · roof 5% · table 3% · moon 2%

🎯 Predict modespeed1×

Guided practice

guided 1

Implement the decoder zoo

25 min

Create decoding_lab.py importing your Day-99 checkpoint, and paste the starter below.
It implements greedy, temperature, top-k, and top-p sampling as one function with flags.
From the same 20-character seed, generate 200 characters with: greedy; T=0.3; T=0.8; T=1.5; T=0.8 + top_k=20; T=0.8 + top_p=0.9.
Paste all six outputs into a comparison file. Label each with one line: coherence, diversity, failure modes.
Run greedy twice — confirm identical output. Run T=0.8 twice with torch.manual_seed(42) set both times — confirm identical. Remove the seed — confirm different. Write one sentence on what the seed controls.

🐍 python — editable, runs in your browser

import torch
import torch.nn.functional as F
from model import TinyGPT

ckpt = torch.load('tinygpt_ckpt.pt')
chars = ckpt['chars']
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
device = 'cuda' if torch.cuda.is_available() else 'cpu'
block_size = 128

model = TinyGPT(vocab_size=len(chars), block_size=block_size).to(device)
model.load_state_dict(ckpt['model'])
model.eval()

@torch.no_grad()
def sample(seed_text, n=200, greedy=False, temperature=1.0,
           top_k=None, top_p=None):
    idx = torch.tensor([[stoi[c] for c in seed_text]], device=device)
    for _ in range(n):
        logits, _ = model(idx[:, -block_size:])
        logits = logits[:, -1, :]
        if greedy:
            nxt = logits.argmax(dim=-1, keepdim=True)
        else:
            logits = logits / temperature
            if top_k is not None:
                v, _ = torch.topk(logits, top_k)
                logits[logits < v[:, [-1]]] = -float('inf')
            probs = F.softmax(logits, dim=-1)
            if top_p is not None:
                sp, si = torch.sort(probs, descending=True)
                keep = torch.cumsum(sp, dim=-1) <= top_p
                keep[:, 0] = True  # always keep the top token
                sp = sp * keep
                sp = sp / sp.sum(dim=-1, keepdim=True)
                pick = torch.multinomial(sp, 1)
                nxt = si.gather(-1, pick)
            else:
                nxt = torch.multinomial(probs, 1)
        idx = torch.cat([idx, nxt], dim=1)
    return ''.join(itos[i] for i in idx[0].tolist())

seed_text = 'Once more unto the '
for name, kw in [
    ('greedy', dict(greedy=True)),
    ('T=0.3', dict(temperature=0.3)),
    ('T=0.8', dict(temperature=0.8)),
    ('T=1.5', dict(temperature=1.5)),
    ('T=0.8 top_k=20', dict(temperature=0.8, top_k=20)),
    ('T=0.8 top_p=0.9', dict(temperature=0.8, top_p=0.9)),
]:
    print('=== ' + name + ' ===')
    print(sample(seed_text, **kw))
    print()

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Visualize one decision

15 min

Pick one generation step: feed a seed, grab the final-position logits, and compute softmax probabilities at T = 0.3, 1.0, and 1.5.
Print the top-10 tokens with their probabilities at each temperature, side by side.
Observe how the mass concentrates when cold and spreads when hot — this table IS the sampling-dist visual, built by you.
Compute the entropy of each distribution with -(p * p.log()).sum() over nonzero entries and confirm entropy rises with temperature (Day 62, quantified).

On your own

The repetition-loop clinic

20 min

Force a failure, then fix it. (1) Find a seed and settings where greedy decoding enters a repetition loop within 300 characters (low-loss checkpoints loop more readily — try a seed containing a phrase common in the corpus). (2) Fix it two independent ways without changing the model: via sampling settings, and via a repetition penalty you implement yourself (divide the logits of the last 50 emitted token ids by 1.3 before softmax). (3) Write a 5-line "diagnosis card": symptom, mechanism (why does greedy loop? think feedback), fix A, fix B, trade-off of each.

Hints: the loop mechanism is that repetition raises the probability of further repetition — a feedback cycle only randomness or penalties break. Compare your fixes' side effects on coherence.

Ship before you stop

Decoding cheat sheet for your future self

Write decoding-cheatsheet.md: for each of five real tasks — data extraction, code generation, marketing copywriting, customer-support answers, brainstorming — recommend decoding settings (or "provider-tuned; steer via prompt" where APIs expose no knobs), justified in one sentence each by today's mechanics. Include your six labeled tiny-GPT samples as an appendix, the temperature/entropy table from guided 2, and the repetition diagnosis card. Commit it — you will consult this on Day 107 (API cost/latency tuning) and Day 155 (serving parameters in vLLM).

Rubric — check what you completed (0/6)

All five tasks mapped to settings with mechanistic (not vibes) justificationsSix labeled samples included, generated from the same seedEntropy-vs-temperature table computed, showing entropy rising with TRepetition loop reproduced and fixed two ways, with trade-offs statedNotes correctly state where sampling knobs do and do not exist (local vs some frontier APIs)Committed to the notes repo

Common mistakes & misconceptions

Saying "the model chose the word." The model outputs a distribution; the decoder chooses. Keeping these separate makes half of LLM debugging tractable.
Treating temperature 0 / greedy as guaranteeing identical outputs from hosted APIs. Batching and floating-point non-determinism mean providers rarely promise bitwise reproducibility — design for variance.
Cranking temperature to "make it smarter." Temperature adds entropy, not intelligence; for reasoning tasks high T mostly adds errors.
Using top-k with a fixed k as if distributions had constant shape. When the model is certain, k=50 admits junk; when uncertain, it may exclude good options — that adaptivity is exactly why top-p exists.
Forgetting to renormalize after truncating the distribution (top-k/top-p) — probabilities must sum to 1 before multinomial sampling.
Diagnosing repetition loops as a training failure. They are usually a decoding failure — greedy/low-T feedback — fixable in seconds with sampling or penalties.

Knowledge check

Q1. Temperature is applied by…

Q2. Top-p is preferred over top-k in many defaults because…

Q3. A support bot keeps emitting the same sentence in a loop. Most likely first fix?

Go deeper — curated resources

articleThe Illustrated GPT-2 — how generation works, visually ↗30 min courseHugging Face LLM Course — generation & decoding strategies ↗20 min bookDive into Deep Learning — sequence generation chapters ↗20 min

If you have a third hour

The Curious Case of Neural Text Degeneration (Holtzman et al. 2019) ↗ — The paper that introduced nucleus sampling and documented why maximization-based decoding produces degenerate text.

Done means

Six labeled samples generated from one seed across strategies
Entropy shown rising with temperature, computed numerically
Repetition loop reproduced and fixed two independent ways
Cheat sheet committed with all five task recommendations
Quiz ≥ 2/3

How this connects

← Back: The softmax you temperature-scale is the same function from Day 94's attention; the entropy you computed quantifies Day 62's "surprise as a currency." Your Day-99 generate() was a naive decoder — today you made it a toolbox.

Forward →: Day 103 runs open models where these knobs are yours again; Day 107 covers steering hosted APIs where they often are not. Day 139 turns "outputs vary across runs" into proper statistics for evals, and Day 155's vLLM serving exposes exactly these parameters at scale.

Unlocks: D103 The Model Landscape & Local Inference · D105 Week 15 Checkpoint: Phase 5 Assessment