Day 127 · Teaching the polymath your house style

Fine-Tuning I — When & Data

You will be able to

Decide between prompting, RAG, and fine-tuning for a given requirement using an explicit decision tree
State what supervised fine-tuning reliably changes (form, style, format) and what it does not (fresh knowledge)
Build a small SFT dataset in chat-JSONL format and validate it programmatically
Detect contamination and leakage between training data and eval data before spending money
Estimate hosted fine-tune costs and defend the spend against a prompt-engineering baseline

Today's ~120 minutes

Spaced-rep warm-up: due cards from Week 18 (agents, MCP)	10 min
ELI5 + tech read; sketch the decision tree from memory	20 min
Guided: decision gauntlet + build/validate the SFT dataset	40 min
Practice: audit the flawed dataset	20 min
Project: fine-tune decision memo for the capstone	20 min
Quiz + flashcards	10 min

Builds on: Day 100 — How LLMs are trained (SFT, RLHF) · Day 109 — Prompt engineering II — A/B on cases · Day 113 — RAG architecture — why RAG for knowledge

The analogy

🧸 ELI5⚙ Tech

Your polymath consultant already speaks fluent everything. You have three ways to shape their work. A briefing memo (the prompt) tells them what you want today — cheap, instant, revisable. An open-book exam (RAG) hands them your company documents so they can cite facts they never memorized. Finishing school (fine-tuning) drills them for weeks until your house style becomes muscle memory — the exact report format, the exact tone, no memo needed.

Notice what finishing school does NOT do: it does not teach them this quarter's numbers. Drilling changes how someone behaves, not what happened in the world last Tuesday. That is the single most common fine-tuning mistake in industry: paying for finishing school when what you needed was to hand over the binder. Fine-tune for form and reflexes; retrieve for facts; prompt for everything you might want to change tomorrow.

Why this matters on the job

"Can't we just fine-tune it on our data?" is the question every FDE hears in the first customer meeting, and the correct answer is usually "not yet, and maybe never." Teams burn weeks and real money fine-tuning for knowledge that RAG would have delivered in a day — and the tuned model still hallucinates because facts were never the fixable part. Being the person who can draw the decision tree on a whiteboard, name what SFT actually buys (format compliance, tone, latency via shorter prompts), and demand evals before and after is a hiring signal and a customer-trust signal at once.

Watch it happen

Teaching the polymath your house style — the adaptation decision tree

step 1 / 5

1. Promptinghours, ~free→2. RAGdays→3. Fine-tuneweeks, data, evals

✉ better briefing memo? few-shot examples? output contract? (Days 108–110)

A team says "the model isn't good enough — let's fine-tune!" Stop. There is an ordered checklist, and fine-tuning is LAST. First gate: have you exhausted prompting?

🎯 Predict modespeed1×

Guided practice

guided 1

The decision-tree gauntlet

15 min

For each of the eight scenarios below, write one line: PROMPT, RAG, FT, or a combination — plus the deciding factor.
Scenarios: (a) support bot must answer from a policy wiki updated weekly; (b) model must emit your exact 12-field incident-report JSON, and few-shot gets 82% valid while you need 99%; (c) legal team wants answers "in our firm's voice" across thousands of interactions; (d) chatbot doesn't know your product launched last month; (e) you want a small cheap model to match the big model's quality on ONE narrow classification task; (f) output must never include competitor names; (g) a demo tomorrow needs friendlier tone; (h) medical summarizer must use your clinic's section headers, and the 1,800-token instruction prompt is blowing the latency budget.
Check yourself: (a) RAG — facts change weekly. (b) FT candidate — format reliability after prompting plateaued; also try schema-constrained output from Day 110 first. (c) FT — persistent style at scale. (d) RAG — knowledge. (e) FT/distillation — Day 129 territory. (f) guardrail/output filter, not FT — FT gives you no hard guarantee. (g) PROMPT — tomorrow! (h) FT — bake the instructions into weights to shrink the prompt.
Score yourself out of 8 and note which distinctions you missed.

guided 2

Build and validate an SFT dataset

25 min

Create sft_dataset.py with the starter code. It defines six raw support-ticket answers in your "house style" (terse, empathetic opener, numbered steps, escalation footer).
Convert them to chat-JSONL: one line per example, each with a short system message, the user ticket, and the ideal assistant reply. Write train.jsonl.
Run the validator: it checks JSON parses per line, roles alternate correctly, no assistant turn is empty, and flags near-duplicate user turns (Jaccard word overlap > 0.8).
Now poison it deliberately: add a duplicate, an example with a rude reply, and an example whose user text equals one of your five held-out eval tickets. Confirm the validator catches the first and third — and notice it CANNOT catch the second. Quality review is human work.
Write eval_holdout.jsonl with the five held-out tickets and confirm zero overlap with training inputs.

🐍 python — editable, runs in your browser

import json
from itertools import combinations

def jaccard(a: str, b: str) -> float:
    wa, wb = set(a.lower().split()), set(b.lower().split())
    return len(wa & wb) / max(1, len(wa | wb))

def validate(path: str, eval_path: str | None = None) -> list[str]:
    problems, rows = [], []
    for i, line in enumerate(open(path), 1):
        try:
            row = json.loads(line)
        except json.JSONDecodeError:
            problems.append(f"line {i}: not valid JSON"); continue
        msgs = row.get("messages", [])
        roles = [m.get("role") for m in msgs]
        if not msgs or roles[-1] != "assistant":
            problems.append(f"line {i}: must end with an assistant turn")
        if any(m.get("role") == "assistant" and not m.get("content", "").strip()
               for m in msgs):
            problems.append(f"line {i}: empty assistant content")
        rows.append((i, row))
    users = [(i, m["content"]) for i, r in rows
             for m in r["messages"] if m["role"] == "user"]
    for (i, a), (j, b) in combinations(users, 2):
        if jaccard(a, b) > 0.8:
            problems.append(f"lines {i},{j}: near-duplicate user turns")
    if eval_path:
        eval_users = {m["content"].strip().lower()
                      for line in open(eval_path)
                      for m in json.loads(line)["messages"]
                      if m["role"] == "user"}
        for i, u in users:
            if u.strip().lower() in eval_users:
                problems.append(f"line {i}: CONTAMINATION — appears in eval set")
    return problems

if __name__ == "__main__":
    issues = validate("train.jsonl", "eval_holdout.jsonl")
    print("\n".join(issues) if issues else "dataset clean")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Audit the flawed dataset

20 min

Write flawed.jsonl yourself: 10 SFT examples for a "polite refund-request classifier" where you deliberately plant five distinct data diseases — (1) an inconsistent label (same situation, opposite answers), (2) a leaked eval case, (3) a formatting inconsistency (one reply in JSON, rest in prose), (4) a factually wrong but fluent answer, (5) a class imbalance (8 of 10 examples are one label).

Then write audit.md: for each disease, state what a model trained on it would learn and which of the five your Day-127 validator can catch automatically versus which need human review. Finish with a one-line rule for each disease that you would put in a dataset checklist.

Hints: think about what SFT optimizes — it clones the distribution you give it, including its bugs. Which diseases change the distribution silently?

Ship before you stop

Fine-tune decision memo for the capstone

Write docs/ft-decision.md in your capstone repo. Part 1: apply the decision tree to your Docs-QA service — should any component be fine-tuned? (Likely answer: the answer-generation prompt stays prompted + RAG, but argue it honestly, including the "shrink the mega-prompt" angle.) Part 2: pick the strongest hypothetical FT candidate in your system (e.g., a citation-formatting or query-rewriting model) and write its dataset spec: source of examples, target size, chat-JSONL schema, review process, contamination checks against your future golden set (Day 134), and a cost estimate versus the prompting baseline. This memo is the "should we fine-tune?" answer you will give customers as an FDE.

Rubric — check what you completed (0/6)

Decision tree applied to at least two capstone components with the deciding factor named for eachCorrectly separates knowledge gaps (RAG) from behavior gaps (FT candidates)Dataset spec includes format, size target, review step, and an explicit contamination checkCost estimate compares training + inference pricing against the long-prompt baseline with real numbersStates the eval that must exist before any tune is runCommitted to the capstone repo

Common mistakes & misconceptions

Fine-tuning to add knowledge. SFT teaches behavior; facts seen once or twice are unreliable, and they go stale. Use RAG for knowledge, FT for form.
Skipping the prompting baseline. If few-shot plus an output contract reaches target, the correct amount of fine-tuning is zero — and you can change course tomorrow.
Worshipping dataset size. 300 reviewed examples beat 30,000 scraped ones; SFT clones your data's flaws with perfect fidelity.
No before/after eval. Without a held-out score you cannot see the tune's regressions — models routinely get better at your task and quietly worse at everything adjacent.
Contaminated evals: an eval question (or close paraphrase) sitting in training data inflates scores. Check overlap programmatically, like the Day 69 leakage discipline.
Forgetting tuned-model economics: training cost is visible, but higher per-token inference pricing and being pinned to a model snapshot are the costs that bite later.

Knowledge check

Q1. A customer's chatbot keeps answering questions about their product wrongly because the product changed last quarter. Best first move?

Q2. Which outcome is supervised fine-tuning MOST reliable at delivering?

Q3. Your fine-tuned model scores 96% on your eval set, up from 71%. Before celebrating, the FIRST thing to check is…

Go deeper — curated resources

courseHugging Face LLM Course — fine-tuning chapters ↗30 min articleChip Huyen — blog (RLHF/SFT & LLM engineering posts) ↗25 min repoOpenAI Cookbook — fine-tuning examples ↗20 min

If you have a third hour

DPO and preference tuning — SFT clones demonstrations; DPO-style preference tuning learns from chosen-vs-rejected pairs. Useful when "better" is easier to compare than to write. Day 100 sketched where it sits in the pipeline.

Done means

Decision gauntlet scored ≥ 6/8 with reasons written
train.jsonl passes the validator; contamination case caught
Flawed-dataset audit names all five diseases and which are machine-catchable
ft-decision.md committed to the capstone repo
Quiz ≥ 2/3

How this connects

← Back: Day 100 showed SFT inside the training pipeline — today you decide when to buy a run of it. The contamination check is Day 69's leakage discipline, and the "prompt first" rule is Day 109's A/B habit.

Forward →: Tomorrow (Day 128) you run an actual LoRA fine-tune and judge it honestly. Day 129 covers when a distilled or smaller model wins. The "evals before and after" demand becomes your whole job on Days 134–140.

Unlocks: D128 Fine-Tuning II — LoRA Lab · D133 Week 19 Checkpoint: Red-Team Your RAG