Distillation, Quantization & Model Selection
- Explain knowledge distillation and why soft teacher outputs carry more signal than hard labels
- Quantize a weight tensor by hand and measure the rounding error you traded for memory
- Name the standard quantization levels (fp16/bf16, int8, 4-bit) and their typical quality impact
- Build a latency/cost/quality selection matrix and use it to pick a model per task
- Design a cascade/router that sends easy traffic to a small model and hard traffic to a big one
| Spaced-rep warm-up: due cards (LoRA math, FT decision tree) | 10 min |
| ELI5 + tech read; memory math for 7B at four precisions on paper | 20 min |
| Guided: hand quantization lab + selection matrix | 40 min |
| Practice: design the triage cascade | 20 min |
| Project: model-selection worksheet | 20 min |
| Quiz + flashcards | 10 min |
Builds on: Day 103 — The model landscape & local inference · Day 107 — Cost math & token budgets · Day 128 — LoRA — adapting instead of replacing
You would not hire a Nobel laureate to sort your mail. Some tasks need the giant brain; most need a competent, fast, cheap one — and the art is matching brain to task. There are two ways to get a right-sized brain. Distillation is apprenticeship: the master (big model) works through thousands of examples while the apprentice (small model) studies not just the master's final answers but their hesitations — "it's probably a 7, but it could be a 1" — and that nuance teaches faster than bare answers ever could. Quantization is note-taking with fewer decimal places: the brain's connection strengths get stored as rougher numbers. Round 3.14159 to 3.14 and almost nothing changes; round it to 3 and mistakes creep in. Half the storage, nearly the same mind — down to a point, and past that point, a cliff.
Then comes the dispatcher's trick: don't pick ONE brain. Let the mail sorter handle everything, and escalate to the laureate only when the sorter is unsure.
"Which model should we use?" is the second question in every customer engagement (right after "can we fine-tune?" — Day 127). The naive answer — the biggest one — dies on contact with the invoice: at production volume, a frontier model on every request can cost 30–100× more than a routed mix with no measurable quality difference on the easy 80% of traffic. FDEs who can show a selection matrix with real numbers, and a cascade design that protects quality where it matters, turn a pricing objection into a closed deal. This is also exactly the Day 156 cost-engineering groundwork.
Guided practice
Quantize a tensor by hand
20 min- Create
quant_lab.pywith the starter and run it. - It quantizes a normally-distributed weight tensor to int8 and int4 with a single per-tensor scale, then reconstructs and prints mean absolute error.
- Now the outlier experiment: it plants a few large outlier weights (as real LLM layers have) and re-quantizes. Watch per-tensor error jump — the outliers stretched the scale and everyone else lost precision.
- Fix it with per-group scales (groups of 64) and confirm the error drops back down. You have just reimplemented the core idea behind modern 4-bit schemes.
- Record the memory math in a comment: bytes per weight at fp32/fp16/int8/int4, and total for a 7B-parameter model at each (28 GB / 14 / 7 / 3.5).
Build the selection matrix with real numbers
20 min- Create
selection.py. Define four candidate "models" as dicts with published-style pricing and latency: a frontier model (in ~3.00/out ~15.00 per MTok, 900ms TTFT), a mid model (0.80/4.00, 500ms), a small hosted model (0.10/0.40, 250ms), and a local quantized 8B (electricity-only ~0.02 effective, 400ms on your hardware, privacy: on-prem OK). - Define your capstone's answer workload: 1,200 requests/day, ~2,500 input tokens (question + retrieved chunks), ~350 output tokens.
- Compute monthly cost per model. Print the matrix: model, $/month, TTFT, quality column left as "TBD — Day 134 eval", constraints.
- Answer in comments: which models are even eligible if the customer requires no data leaves their VPC? Which would you pilot first and what eval evidence would change your pick?
- Add a cascade row: 80% of traffic to small, 20% to frontier — compute the blended cost and compare.
On your own
Design the triage cascade
20 minDesign (on paper + pseudocode, no API calls needed) a two-tier cascade for the Day 126 support-triage agent: a small model classifies and drafts; hard tickets escalate to the big model.
You must decide: (1) the routing signal — pick TWO candidates (e.g., small model outputs a confidence field; a rule on ticket length/category; small model's self-check "is this answer well-grounded?") and state each one's failure mode; (2) what happens when the router is wrong in each direction (cheap-model answer shipped on a hard ticket vs expensive escalation of an easy one) and which error is more costly for THIS product; (3) the eval you would run to set the escalation threshold, and the metric it optimizes (cost at fixed quality, or quality at fixed cost).
Hints: self-reported confidence is poorly calibrated — Day 135 shows judges have the same disease. Asymmetric error costs should set the threshold, not symmetry.
Model-selection worksheet for the capstone
Create docs/model-selection.md in the capstone repo: a reusable worksheet, filled in once for your Docs-QA service. Sections: (1) workload profile (requests/day, token shapes — measured from your D119 logs if you have them, estimated otherwise); (2) the candidate matrix from guided work, with a quality column explicitly marked "pending Day 134 golden set" — do not fake numbers; (3) constraint gates (privacy, context length, structured-output support) that eliminate candidates before price is considered; (4) the cascade proposal with blended-cost math and its router-eval plan; (5) a decision: what the capstone runs TODAY and the evidence that would change it. This worksheet is a template you will reuse in every future engagement.
Common mistakes & misconceptions
- Picking models from public leaderboards. Benchmark rank on MMLU does not transfer to YOUR task; the only quality column that counts comes from your eval set (Day 134).
- Treating quantization as free. fp16 and good int8 effectively are; 4-bit usually costs a little and sometimes a lot on specific capabilities (math, long-tail languages). Eval the quantized artifact, not the original.
- One scale for the whole tensor. Outlier weights wreck per-tensor quantization; per-channel/per-group scales are why modern schemes work — you proved it in the lab.
- Distilling for breadth. Distillation transfers a slice of behavior onto your task distribution; expecting the student to generalize like the teacher everywhere ends in disappointment.
- Building a cascade and never evaluating the router. A miscalibrated router silently ships small-model answers to hard queries — the failure is invisible without trajectory-level logging.
- Comparing models at different decoding settings or prompt formats — you measured the harness, not the models.
Q1. Why do soft teacher probabilities ("70% cat, 25% fox, 5% car") teach a student more than the hard label "cat"?
Q2. A 7B model quantized from fp16 to 4-bit needs roughly how much memory for weights?
Q3. Your per-tensor int4 quantization has terrible error, but per-group scales fix it. What was the cause?
Go deeper — curated resources
- GPTQ and AWQ papers ↗ — Post-training quantization that minimizes layer output error rather than weight error — why calibration data makes 4-bit usable.
- Quantization lab run; outlier effect and per-group fix observed and explained
- Selection matrix computed with correct monthly cost math
- Cascade design names two routing signals with failure modes and an eval plan
- model-selection.md committed to the capstone repo
- Quiz ≥ 2/3
← Back: Temperature-softened distributions are Day 62's softmax story put to work; the "small model per narrow task" logic extends Day 127-128's adaptation theme; the pricing arithmetic is Day 107's cost math at portfolio scale.
Forward →: The quality column you left "TBD" gets filled by the Day 134 golden set and Day 136 RAG metrics. Day 155-156 turn quantization's bandwidth story into serving optimization, and the router idea becomes production model-routing on Day 156.