Structured Outputs — Forms, Not Essays
- Explain why free-text LLM output breaks pipelines and what schema-constrained output guarantees
- Design extraction schemas with pydantic: enums for closed sets, nullable fields for honest absence
- Build a validate-and-repair loop that feeds validation errors back to the model
- Use native schema-constrained output (client.messages.parse) and compare it with prompt-based JSON
- Measure extraction failure rates over a golden set instead of trusting single successes
| Spaced-rep warm-up: due cards (A/B, contracts) | 10 min |
| Tech read: the five-layer contract stack | 20 min |
| Guided: schema + validation + repair loop | 30 min |
| Guided: native mode head-to-head | 20 min |
| Practice: golden set + failure-rate report | 25 min |
| Project packaging + quiz + flashcards | 15 min |
Builds on: Day 108 — Output contracts in prompts · Day 68 — pydantic data contracts · Day 107 — The client wrapper
Ask a brilliant intern to "look through this contract and tell me about it" and you get an essay — insightful, differently organized every time, useless to a computer. Hand the same intern a form — Party names: ___, Effective date: ___, Auto-renews: yes/no, Termination notice days: ___ — and you get something a filing system can swallow. Same intern, same contract; the difference is the form.
LLM pipelines live or die on this. The model's natural output is essay; your database, your billing system, and the next function in your pipeline eat forms. So you do three things. Design the form carefully: multiple-choice boxes where only certain answers are legal (enums), and an explicit "not stated" option so the intern never invents a date to fill a blank (nullable fields). Check every submitted form against the rules before accepting it (validation). And when a form comes back wrong, do what a good office does: return it with the errors circled and ask for a corrected copy (the repair loop). Modern APIs add a marvel on top — a mode where the intern is physically unable to write outside the boxes. You will use both, because even a perfect box can contain a wrong answer.
Extraction is arguably the most deployed LLM use case in industry: invoices, tickets, resumes, contracts, logs — unstructured in, database rows out. It is also where "works in the demo" dies fastest: a 2% malformed-output rate is invisible in ten manual tests and a nightly pipeline explosion at ten thousand documents. The schema-validate-repair-measure pattern you build today is the backbone of your Day-112 graded assessment, reappears in every agent's tool arguments (Day 111), and structures the citation output of your capstone (Day 119). Failure-rate thinking — measuring per-field over a golden set — is the eval mindset arriving early.
Guided practice
Schema, extraction, validation, repair
30 min- Create
extract.pywith the starter: a support-ticket schema and a prompt-based extractor with a repair loop. - Run it on the three sample documents. Inspect outputs and the repair counter.
- Sabotage test: change the schema so urgency adds a value the prompt does not mention (e.g. 'critical'), and watch validation catch model outputs; then fix.
- Absence test: doc 3 has no order id — confirm the model emits null rather than inventing one. If it invents, strengthen the null instruction and note the wording that worked.
- Log every attempt (valid or not) to JSONL via your Day-107 wrapper habits.
Native structured output, head to head
20 min- Add
extract_nativeusing client.messages.parse with output_format=Ticket (per the structured-outputs docs) — the schema is enforced at generation time and resp.parsed_output returns the validated instance. - Run both extractors over the same three docs. Compare: validity failures (native should have none), VALUES (do the two modes ever disagree on urgency or nulls?), and latency from your logs.
- Write a three-line comparison: what native mode guarantees, what it cannot guarantee, and when you would still keep the repair loop (hint: OpenAI-compatible backends without parse support; semantic repair on wrong-but-valid values).
- Note the dialect equivalent in your notes: OpenAI-shape APIs express this as a response_format json_schema parameter.
On your own
Golden set + failure-rate report
25 minBuild a 10-document golden set for the ticket schema: write or adapt 10 varied support emails, and label expected values for every field yourself (the labels ARE the hard part — notice the judgment calls and write them down).
Then run both extractors over all 10 and produce a small report: schema-validity rate, per-field accuracy against your labels, null-precision (invented values vs correct nulls), and repair count. State which extractor you would ship for this task and your single biggest error class.
Hints: score lists (issues) leniently — count a hit if the key issue is captured; ambiguous labels are a finding, not a nuisance (Day 134 calls this labeling guidelines); 10 docs is small — say "suggests", not "proves".
extractor.py — the reusable extraction module
Refactor today into a toolkit module: extractor.py exposing extract(doc, schema_class, system_extra=None) — generic over any pydantic model, building the JSON-shape instructions from the schema automatically (model_json_schema() helps), with the repair loop, attempt logging, and a native-mode flag. Include the ticket schema as one example plus a second schema of your choice (e.g. invoice or meeting-notes). Ship the golden set and report.py that prints the failure-rate table. README documents the five-layer contract stack. This module is graded on Day 112 against a fresh golden set, and the capstone's citation extraction (Day 119) imports it.
Common mistakes & misconceptions
- Trusting a demo that parsed once. Malformed-output rates of a few percent are invisible in manual testing and fatal at pipeline scale — measure over a set.
- Making every field required. Required fields on optional facts force the model to fill boxes — you are mandating hallucination. Optional + "use null" is the honest design.
- Parsing with regex or string-slicing instead of a real validator. pydantic gives you typed errors you can feed back; a regex gives you silent corruption.
- Believing schema enforcement equals correctness. Native mode guarantees shape, not truth — the wrong-but-valid enum survives it. Golden-set accuracy is the real metric.
- Repairing forever. Uncapped repair loops burn money on a document the schema fundamentally does not fit; cap at 2-3 and route failures to a dead-letter queue for humans.
- Letting markdown fences sneak in. Models love wrapping JSON in code fences; say "raw JSON only, no fences" and strip defensively before parsing.
Q1. Why are Optional fields with a "use null" instruction an anti-hallucination lever?
Q2. Native schema-constrained output (messages.parse / json_schema modes) guarantees…
Q3. The repair loop works by…
Go deeper — curated resources
- Streaming partial JSON — UIs sometimes render extraction results as they stream. Investigate partial-JSON parsing (incremental parsers tolerate incomplete objects) and why strict validators cannot run until the stream closes.
- Prompt-based extractor with repair loop runs on all samples
- Native parse mode compared with values-level diffs noted
- Golden set labeled and failure-rate report produced
- Null-handling verified: no invented values on absent facts
- extractor.py committed generically over two schemas
- Quiz ≥ 2/3
← Back: This is Day 68's pydantic data-contract discipline aimed at model output, wrapped in Day 108's output-contract prompting, and measured with Day 109's suite mindset.
Forward →: Tool arguments on Day 111 are schema-constrained outputs by another name. Day 112 grades your extractor on a fresh golden set; Day 130 extends extraction to documents with vision; and the capstone's cited answers (Day 119) are structured outputs with a citations field.
Unlocks: D111 Tool Use & Function Calling · D112 Week 16 Checkpoint: The Prompt Lab · D130 Multimodal I — Vision & Documents · D131 Multimodal II — Audio & Voice