How LLMs Are Trained
- Describe the three-stage pipeline: pretraining, supervised fine-tuning, preference tuning (RLHF/DPO)
- Explain what each stage adds and what it cannot add
- State why assistant models refuse some requests and where that behavior comes from
- Place the system prompt correctly: steering at inference time, not training
- Estimate the rough scale of data and compute involved in a frontier pretraining run
| Spaced-rep warm-up: due cards incl. Day-99 training cards | 10 min |
| ELI5 + tech read; study the llm-pipeline visual | 25 min |
| Guided: map the pipeline for a real model | 20 min |
| Guided: base-model behavior with your tiny GPT | 20 min |
| Practice: customer-question gauntlet | 20 min |
| Project: pipeline explainer + quiz + flashcards | 25 min |
Builds on: Day 99 — Training the tiny GPT · Day 95 — The Transformer architecture · Day 62 — Cross-entropy loss
Imagine raising a polymath. Stage one: for years they read everything — libraries, newspapers, code, forum arguments — not to memorize but to get so good at predicting how any text continues that they absorb grammar, facts, styles, and reasoning patterns along the way. At the end they are astonishingly knowledgeable but strange company: ask them a question and they might continue it with three more questions, because that is what text on the internet often does.
Stage two: a short finishing school. You show them a few thousand examples of the format you want — question in, helpful answer out — and they learn to behave like an assistant.
Stage three: a panel of coaches. The polymath writes several answers; coaches rank them; the polymath learns to produce more of what gets ranked highly — more helpful, more honest, less harmful. That is the whole pipeline: read everything (pretraining), learn the job (supervised fine-tuning), then be coached on taste and judgment (preference tuning). Your Day-99 tiny GPT did stage one only, in miniature — which is exactly why it imitates Shakespeare but cannot answer a question.
Customers constantly ask "can we train it on our data?" and "why did it refuse that?" — both answers live in this pipeline. Knowing that knowledge comes overwhelmingly from pretraining, format from SFT, and refusals from preference tuning tells you what fine-tuning can and cannot fix (Day 127's decision tree), why RAG beats retraining for fresh knowledge (Day 113), and why a system prompt steers behavior but cannot add facts. This mental model is also a standard LLM-fundamentals interview question.
Raising a polymath — pretraining to assistant, stage by stage
step 1 / 5Stage 1 — pretraining: months of "guess the next token" on trillions of tokens of text and code. Out comes a BASE model: brilliant autocomplete, zero manners.
Guided practice
Map the pipeline for a model you use
20 min- Pick a model you actually use (e.g. Claude, or an open model like Llama). Using its official model card / release notes, fill in a table with one row per pipeline stage: objective, data type, data scale (order of magnitude), what it contributes.
- For each stage, write the one thing it CANNOT do (e.g. SFT cannot add knowledge that pretraining never saw).
- Base vs instruct: many open models ship both variants. Find one pair on Hugging Face (e.g. a base and an -Instruct release of the same model) and note the difference the card describes.
- Add a final row for "inference-time steering" (system prompt) to cement that it is not a training stage.
Feel the base-model difference
20 min- Your Day-99 tiny GPT is a pure base model. Prompt it with a question formatted as a question: encode "What is thy name?" as the seed and generate 200 characters.
- Observe: it continues in corpus style — it does not "answer". Write down what it produced.
- Now write the SFT dataset you would need to make it answer: five example dialogues in a consistent format (e.g. Q: ... A: ...). You will not train on them today — designing the data IS the exercise.
- In three sentences, explain why five examples of format can change behavior when the knowledge was already in the weights — and what would happen if you asked it about something absent from the corpus.
On your own
The customer-question gauntlet
20 minAnswer these four realistic customer questions in 2-4 sentences each, using pipeline vocabulary precisely:
(1) "Can we fine-tune the model so it knows our 2026 product catalog?" (2) "Why does it refuse to write our penetration-testing report? Can we turn that off?" (3) "If we put our style guide in the system prompt, is the model learning it?" (4) "Why does the API model sound so different from the open base model we downloaded?"
Constraint: each answer must name the relevant pipeline stage. Hints: (1) knowledge vs format — foreshadow RAG; (2) preference tuning creates dispositions, not toggles; (3) inference vs training; (4) SFT + preference tuning.
Pipeline explainer diagram
Create llm-training-pipeline.md in your notes repo: a text/ASCII diagram of the three stages plus inference-time steering, and under each stage exactly three bullets — what goes in, what it costs (order of magnitude), what it adds. Then a "myths" section debunking three misconceptions you might hear from a customer (pick from the practice gauntlet). Keep it to one page — this is the skeleton of your Day-105 stakeholder explainer, and you will reuse it verbatim in the Day-127 fine-tuning decision conversation.
Common mistakes & misconceptions
- Believing the system prompt trains the model. It is input tokens at inference time — the model weights never change during your API calls.
- Thinking SFT adds knowledge. It mostly adds format and persona; facts come from pretraining. Missing knowledge needs retrieval (Day 113) or continued pretraining, not a few thousand chat examples.
- Treating refusals as a hardcoded keyword filter. They are learned dispositions from preference data, which is why wording changes results and why jailbreaks exist (Day 132).
- Assuming the reward model is the deployed model. The reward model only scores candidates during training; it is discarded at inference.
- Confusing DPO with RLHF-minus-quality. DPO is a simpler optimization of a similar objective on preference pairs — an implementation choice, not a weaker goal.
- Extrapolating from your tiny GPT that LLMs "just memorize." At scale, compression of trillions of tokens into far fewer parameters forces generalization — memorization cannot fit.
Q1. A customer wants the model to know their internal 2026 pricing. Which pipeline fact matters most?
Q2. Why does a base model often respond to a question with more questions?
Q3. In RLHF, what does the reward model do?
Go deeper — curated resources
- Constitutional AI / RLAIF — Anthropic-style training replaces some human preference labels with AI feedback guided by an explicit set of principles — worth reading about before Day 132's safety discussion.
- Pipeline table completed for a real model with base-vs-instruct noted
- All four gauntlet answers name the correct stage
- Pipeline explainer committed (one page, three myths)
- Quiz ≥ 2/3
← Back: Day 99 was stage one in miniature — you now know precisely what your tiny GPT is missing and why. The cross-entropy objective from Day 62 is the loss for both pretraining and SFT.
Forward →: Day 101 asks what happens as stage one scales. Day 106 starts calling models that went through all three stages. Day 127 turns today's "what can each stage add" into the prompt-vs-RAG-vs-fine-tune decision tree, and Day 132 revisits refusals from the attacker's side.
Unlocks: D101 Scaling Laws, Capabilities & Limits · D103 The Model Landscape & Local Inference · D105 Week 15 Checkpoint: Phase 5 Assessment · D106 LLM APIs I — The Request