Entropy, Cross-Entropy & KL
- Define information as surprise (−log p) and compute it in bits and nats
- Compute the entropy of a distribution and explain why uniform maximizes it
- Explain cross-entropy as "average surprise when betting with the wrong forecast" and why it is THE training loss
- Compute KL divergence and state its three key properties (≥ 0, zero iff equal, asymmetric)
- Convert a language-model loss to perplexity and interpret both against a random baseline
| Spaced-rep warm-up: Days 59–61 cards (log-probs, CIs, p-values) | 10 min |
| ELI5 + tech read, walk the entropy-bits visualizer | 20 min |
| Guided: surprise & entropy + cross-entropy loss lab | 40 min |
| Practice: the temperature dial | 18 min |
| Project: entropy.py + loss-reader's card | 22 min |
| Quiz + flashcards | 10 min |
Builds on: Day 57 — Probability fundamentals · Day 59 — Log-probabilities & the underflow trick
Imagine paying for news in proportion to how surprised you are. "The sun rose this morning" — worthless, you pay nothing. "It snowed in the Sahara" — you would pay a lot. Information theory makes this precise: the surprise of an event is −log of its probability. Certain events carry zero surprise; rare events carry a lot; and surprises ADD — two independent shockers are worth the sum of each (because their probabilities multiply and logs turn products into sums).
Entropy is the average surprise a source dishes out. A fair coin keeps you maximally on your toes: 1 bit per flip. A trick coin that lands heads 99% of the time is boring — almost no surprise per flip. Now the punchline that runs all of deep learning: suppose the world deals events from distribution p, but YOU bet using forecast q. Your average surprise — cross-entropy — is always at least the world's own entropy, and the gap between them, KL divergence, is the exact price of being wrong. Training a model by "minimizing cross-entropy" just means: adjust the forecast until the model is as unsurprised by reality as reality allows. Surprise is the currency; learning is driving down your bill.
Cross-entropy is the loss function you will stare at more than any other number in your career: logistic regression's log-loss (Day 72), every neural net classifier (Days 85–90), and the pretraining objective of every LLM (Day 100) are all cross-entropy. Reading a loss curve without knowing what the units mean — "is 2.3 good?" — is flying blind; today you learn that 2.3 nats on 10 classes IS the random baseline (ln 10). Perplexity in model reports, temperature in sampling (Day 102), and KL penalties in RLHF are all today's vocabulary.
Surprise as a currency — the entropy of a coin, in bits
step 1 / 5A fair coin (p = 0.5) is maximally unpredictable: every flip delivers exactly 1 bit of surprise. That's the peak of this curve.H(p) = −p·log₂p − (1−p)·log₂(1−p) · peak = 1.0 bit
Guided practice
Surprise, entropy, and twenty questions
18 min- Paste the starter. Part A prints the surprise (in bits) of events at p = 0.99, 0.5, 0.1, 0.001 — confirm rare = expensive, and that a 1-in-1024 event costs exactly 10 bits.
- Part B computes coin entropy across biases 0.5 → 0.99. Verify the fair coin maxes out at 1.0 bit and near-certain coins approach 0.
- Part C is the guessing game: 8 equally likely suspects have H = 3 bits — exactly 3 optimal yes/no questions. Now skew the suspects (one has probability 0.65) and watch H drop below 2: a smart guesser asks about the likely suspect first and usually finishes early. Entropy = the unavoidable average number of questions.
- Compute H for a uniform distribution over 50,000 outcomes (a vocabulary-sized die): ~15.6 bits. That is the ceiling a language model starts from before it learns anything.
- Write one line in your notes: "entropy is a property of the ______, cross-entropy is a property of the ______ and the ______." (source; source and forecast)
Cross-entropy as a loss you can feel
22 min- The starter sets up a 4-class problem with 6 labeled examples and four "models": uniform (knows nothing), decent, confident-and-right, confident-and-wrong.
- Compute each model's average cross-entropy loss = mean of −log q(true class), in nats. Predict the ordering before running. Confirm uniform lands at ln 4 ≈ 1.386 — the random baseline for 4 classes.
- Stare at the confident-and-wrong model: ONE example where it put q = 0.001 on the truth costs −ln(0.001) ≈ 6.9 nats, dragging the average above the know-nothing model. Confident wrongness is the cardinal sin — write down why that is exactly the incentive you want a training loss to encode.
- Mini-training: interpolate the uniform model toward the perfect one-hot answers in 10 steps and print the loss at each step — a hand-made loss curve, falling monotonically. This is the shape you will watch on Day 89 for real.
- Compute KL(p_true_freq ‖ q) for each model and confirm KL = cross-entropy − entropy. Then compute both KL directions between two of the models to see the asymmetry with your own eyes.
- Convert each loss to perplexity with exp(loss). The uniform model's perplexity is exactly 4 — "hedging across 4 choices". This number is how LLM papers will speak to you.
On your own
The temperature dial
18 minA model produces logits [4.0, 2.5, 1.0, 0.5, -1.0] over five tokens. Implement a stable softmax with temperature: subtract the max, divide logits by T, exponentiate, normalize.
Your goals: (1) compute the probability distribution and its entropy (in bits) at T = 0.25, 0.5, 1, 2, 100; (2) describe the two limits — as T → 0 the distribution approaches one-hot argmax (entropy → 0, "greedy"), as T → ∞ it approaches uniform (entropy → log₂ 5 ≈ 2.32); (3) in two sentences, connect this to sampling from an LLM: what does raising temperature trade away, and why does T change WHICH token wins never (argmax is invariant) but HOW OFTEN each is sampled always? (4) sample 1,000 tokens at T = 0.5 and T = 2 with rng.choice(p=...) and compare the counts.
Hints: entropy of the T = 1 distribution should land around 1.3 bits. Day 102 runs this exact experiment against a real API — keep your code.
entropy.py + the loss-reader's card
Build entropy.py in your practice repo: entropy(p), cross_entropy(p, q), kl(p, q) (all accepting NumPy arrays, safe against zeros via clipping, with a bits=True/False flag), and perplexity(loss_nats). Add asserts encoding the theory: cross_entropy ≥ entropy, kl ≥ 0, kl(p, p) == 0, and kl asymmetry on an example. Then write loss_card.md — the card you will keep beside every training run: random-baseline loss = ln(k) for k classes (with a table for k = 2, 4, 10, 100, 50,000), what perplexity means, nats vs bits, and the three loss-curve smells (loss above baseline = worse than guessing; loss exploding = confident wrongness or LR too high; train ≪ val = memorizing). Commit both — Day 72 and Day 89 will use this card verbatim.
Common mistakes & misconceptions
- Mixing up bits and nats. log₂ gives bits, ln gives nats; PyTorch cross-entropy reports nats. A "loss of 2.3" on 10 classes is ln 10 — exactly random, not "pretty good".
- Interpreting a loss without its baseline. Always compare to ln(k) for k classes; a loss of 1.0 is great for 50,000 classes and terrible for 2.
- Treating KL as a distance. It is asymmetric — KL(p‖q) ≠ KL(q‖p) — and violates the triangle inequality. Say "divergence", and state the direction.
- Computing log(0). A model that assigns exact zero to an event that happens earns infinite loss and NaNs your training. Clip probabilities (or work in logits) — the same hard-zero problem Laplace smoothing solved on Day 59.
- Believing temperature changes what the model knows. It only reshapes the existing distribution — sharper or flatter; the ranking of tokens is untouched.
- Expecting cross-entropy to reach zero on noisy data. It bottoms out at the data's own entropy H(p) — irreducible uncertainty is not a modeling failure.
Q1. What is the entropy of a fair 8-sided die, and what does the number mean operationally?
Q2. With one-hot true labels, the cross-entropy loss for one example reduces to…
Q3. Your 10-class classifier reports loss 2.30 nats. How is it doing?
Go deeper — curated resources
- Why minimizing cross-entropy = maximum likelihood — The average −log q(data) is exactly the negative log-likelihood; minimizing one minimizes the other. This single identity connects information theory, statistics, and every training run you will launch.
- Entropy computations match theory (fair coin 1 bit, 8 suspects 3 bits, vocab ~15.6 bits)
- Cross-entropy lab run; confident-wrong model's loss explosion observed and explained
- Temperature practice: entropies at five temperatures + sampling comparison done
- entropy.py with passing asserts and loss_card.md committed; quiz ≥ 2/3
← Back: The −log p at the center of everything is Day 59's log-probability trick wearing a crown, and "average over the distribution" is Day 58's expectation. The 1-in-1024 = 10 bits arithmetic is Day 33's halving instinct.
Forward →: Day 72 meets cross-entropy as logistic regression's log-loss; Days 86–89 minimize it with backprop; Day 99 watches your tiny GPT's cross-entropy fall from ln(vocab) as it learns to spell; Day 102 turns the temperature dial on a real model.
Unlocks: D63 Week 9 Checkpoint: Math Assessment · D72 Logistic Regression & Losses · D73 Decision Trees · D94 Attention