Tokenization
- Explain why LLMs use subword units instead of characters or whole words
- Execute the BPE training algorithm by hand and in code on a small corpus
- Use tiktoken to count tokens and estimate the cost of real prompts
- Predict tokenizer quirks: spaces, capitalization, numbers, code, and non-English text
- Connect tokenization to model failures in arithmetic and spelling
| Spaced-rep warm-up: Days 94–95 cards | 10 min |
| ELI5 + tech read; Karpathy tokenizer video (first 40 min at 1.5x) | 30 min |
| Guided: BPE by hand + in code, tiktoken cost lab | 45 min |
| Practice: the failure gallery | 20 min |
| Project commit + quiz + flashcards | 15 min |
Builds on: Day 92 — Embeddings — what tokens become · Day 95 — Transformer inputs & vocab size · Day 5 — Strings & encodings
Before a model can read, someone has to decide what its alphabet is. Whole words? The dictionary never closes — "unfollowable", "COVID-19", "skibidi" — and every unseen word becomes an illiterate shrug. Single characters? Nothing is ever unseen, but "electroencephalography" becomes 25 tiny pieces and the model burns its attention budget just spelling. The working compromise is a Lego kit of subwords: common words are single bricks, rare words snap together from pieces ("token" + "ization"), and any text — any! — can be built, worst case letter by letter.
The kit isn't designed by a linguist; it's grown by counting. Byte-pair encoding starts with raw bytes and repeats one move: find the pair of adjacent pieces that occurs most often in a huge pile of text, glue it into a new brick, add it to the kit. Do that 50,000 times and you have GPT's vocabulary — a frozen record of what was common in its training data. Every quirk you will ever debug traces back to this: " hello" (with space) and "hello" are different bricks, "1234" might be one brick while "1235" is two, and your customer's German compound nouns shatter into gravel. The model never sees letters or words — only your brick IDs.
Tokens are the currency of your career: every API bill (Day 107), every context-window budget (Day 122), and every latency target (Day 155) is denominated in them. An FDE who can say "your prompts average 3,400 tokens, of which 2,100 are a boilerplate header — cache it and costs drop 60%" is worth their rate. Tokenization also explains a whole genus of model failures — miscounted letters, flaky arithmetic, worse performance in Thai than English — and "that's a tokenizer artifact" is a diagnosis you will deliver many times.
The LLM's alphabet — text becomes token IDs
step 1 / 5You type a sentence. The model never sees letters — it sees tokens.
Guided practice
BPE by hand, then in 30 lines
25 min- On paper, train BPE on the corpus "low low low lower lowest" with base vocabulary = characters. Count pairs: (l,o) appears 5 times, (o,w) 5 times… Merge the most frequent (break ties by first seen), write the merge rule, and repeat for 6 merges. Track how "low" becomes one symbol.
- Predict: after your 6 merges, how does the UNSEEN word "lowly" tokenize? (Replay merges in order — you should get [low, l, y] or similar depending on your ties.)
- Now run the starter implementation on the same corpus and compare its merge list to yours. Investigate any difference (tie-breaking order is the usual culprit).
- Encode "lowest lowly slow" with the trained merges and stare at the output: which words are single bricks, which are gravel, and why?
tiktoken lab — count and cost real prompts
20 min- (Local run: pip install tiktoken.) Load the GPT-4-era encoding:
enc = tiktoken.get_encoding('cl100k_base'). - Tokenize and inspect: "hello world", " hello world" (leading space), "Hello World", "HELLO WORLD". Print token counts AND the decoded piece for each ID (enc.decode([id])). Note how case and spaces change everything.
- Tokenize "1234", "12345", "3.14159", and a 6-digit multiplication prompt. Write one sentence connecting what you see to arithmetic flakiness.
- Tokenize the same 2-sentence text in English and (via any translation you have) one non-Latin-script language; compare counts.
- Cost drill: take a realistic 3-paragraph system prompt (write one for a support bot), count its tokens, and compute the monthly cost of sending it on every one of 100k daily requests at a made-up rate of 3 dollars per million input tokens. This number — the price of a wasteful system prompt at scale — is one to remember.
On your own
The tokenizer failure gallery
20 minBuild quirks.md: five demonstrations, each with the tokenization printed and a two-sentence explanation — (1) a spelling task the tokenizer sabotages ("how many r's in strawberry"); (2) an arithmetic case where digit grouping is visibly unhelpful; (3) a trailing-space prompt vs the same without; (4) an English vs non-English cost comparison with the ratio computed; (5) one quirk you DISCOVER yourself by poking around (code, emoji, URLs, and rare names are fertile ground).
Constraints: every claim backed by an actual printed tokenization, not folklore.
Hints: for (5), try tokenizing a UUID, a base64 string, or deeply indented Python.
token_toolkit.py — the counting habit, installed
Commit token_toolkit.py with three functions you will genuinely reuse: count(text, encoding='cl100k_base'); cost(text, price_per_mtok, calls_per_day) returning a daily/monthly estimate; and report(text) printing count, char/token ratio, the 10 longest tokens found, and a fragmentation warning when the char/token ratio drops below 2.5 (a heuristic for "this text tokenizes badly"). Plus your toy BPE from guided 1 as bpe_toy.py with a test asserting the merge list on the low/lower/lowest corpus. Include quirks.md. On Day 107 the cost function grows into your real API cost dashboard; on Day 122 count() budgets agent context windows.
Common mistakes & misconceptions
- Assuming one token = one word. English prose averages ~0.75 words/token; code and non-English can be far worse. Measure with the actual tokenizer.
- Counting tokens with one model's tokenizer and billing another's. Vocabularies differ; for exact billing use the provider's usage fields (Day 106) — tiktoken is for local estimation.
- Ending a prompt with a trailing space and wondering why completions degrade. " world" is its own token; a dangling space pushes the model off its learned distribution.
- Blaming "model stupidity" for arithmetic and spelling failures that are tokenization artifacts — the model literally cannot see digits or letters inside its bricks.
- Forgetting vocab size is a parameter trade-off: 200k vocab shortens sequences but costs vocab·d embedding parameters (yesterday's napkin) and dilutes rare-token training.
- Treating the tokenizer as swappable after training. Model weights are married to their tokenizer's IDs; changing it means retraining or careful surgery.
Q1. Why do LLMs use subword tokenization instead of whole words?
Q2. In BPE training, each iteration…
Q3. A model reliably fails "how many letter r's are in strawberry". The most honest explanation is…
Go deeper — curated resources
- Finish the Karpathy tokenizer video and build minbpe-style byte-level BPE — Extend bpe_toy.py from characters to bytes, train on a page of real text, and encode emoji — watching multi-byte characters split is the moment byte-level BPE clicks.
- Hand-traced BPE merges match the code's merge list (ties explained)
- tiktoken lab run: spaces/case/number quirks observed and noted
- System-prompt cost drill computed and recorded
- quirks.md has five backed demonstrations; token_toolkit.py committed
- Quiz ≥ 2/3
← Back: Tokens are what Day 92's embedding table looks up — vocab size sets its row count (Day 95's vocab·d term). The byte-level guarantees echo Day 5's encodings lesson.
Forward →: Tomorrow your GPT uses the simplest tokenizer (characters) so architecture stays center stage; Day 99 revisits that choice. Day 106 reads token counts off real API responses, Day 107 turns count into cost engineering, and Day 122 budgets agent context in exactly these units.
Unlocks: D97 Tiny GPT Lab I — Build It · D99 Tiny GPT Lab II — Train & Sample · D104 Context Windows & Hallucination Deep-Dive · D106 LLM APIs I — The Request