Day 130 · Eyes for the polymath

Multimodal I — Vision & Documents

You will be able to

Explain how vision-language models turn images into tokens the LLM attends over
Send images to a VLM via API and control token cost by resizing deliberately
Build a document-extraction pipeline: image/PDF in, schema-validated JSON out
Choose between OCR-first and VLM-native pipelines for a given document workload
Name the failure modes of charts, tables, and dense scans — and the checks that catch them

Today's ~120 minutes

Spaced-rep warm-up: due cards (quantization, LoRA)	10 min
ELI5 + tech read; sketch the OCR-vs-VLM routing decision	15 min
Guided: receipt extraction + PDF router	45 min
Practice: the chart interrogation	20 min
Project: receipts-to-ledger extractor with labels	20 min
Quiz + flashcards	10 min

Builds on: Day 110 — Structured outputs & pydantic validation · Day 96 — Tokenization — cost per token · Day 113 — RAG architecture (ingest stage)

The analogy

🧸 ELI5⚙ Tech

Until now your polymath consultant took every job by phone — anything visual had to be described to them in words, badly, by whoever answered the door. Vision-language models give the polymath eyes. Slide a receipt, a chart, or a signed form across the desk and they read it directly: the smudged total, the bar that is taller than the others, the checkbox someone x-ed.

But eyes come with a meter. The polymath doesn't see pixels; the image is diced into small patches, and every patch costs the same way words do — a big photo can cost more than a page of text. And their eyes share the mouth's old habit: when a number is illegible, they will confidently read you A number. So the working rules are: show, don't describe; resize before you slide it over (pay for the detail you need, not the detail you have); and for anything that matters, make them copy figures into a form you can check — never a free-form retelling.

Why this matters on the job

An enormous share of real enterprise value is locked in visual documents: invoices, receipts, contracts, lab reports, dashboards, engineering drawings. "Can it read our PDFs?" comes up in almost every FDE discovery call, and the honest answer is nuanced — native PDFs, scans, and tables each behave differently. Document extraction is also the cleanest possible showcase for the Day 110 discipline: schema out, validate, measure field-level accuracy. Teams that demo free-form "look, it described the invoice!" lose to the ones that show 97% field accuracy on a labeled sample.

Guided practice

guided 1

First sight: image in, schema out

25 min

Find or photograph two receipts (or grab any two receipt images you have rights to). Save as receipt1.jpg, receipt2.jpg.
Create vision_extract.py from the starter: it resizes the image to a sane long edge (1024px), base64-encodes it, and asks the model to fill a pydantic Receipt schema — merchant, date, line items, total — with nulls allowed for unreadable fields.
Run it on both receipts. Validate with pydantic; on validation failure, send the error back for one repair round (Day 110 pattern).
Cross-check: does the sum of line items match the extracted total (within a cent)? Print PASS/MISMATCH — this arithmetic check catches hallucinated digits that no schema can.
Re-run receipt1 at long edge 512 and 2048. Note extraction quality vs the token counts in the usage field — find the knee.

🐍 python — editable, runs in your browser

import base64, io, json
from PIL import Image
from pydantic import BaseModel, ValidationError
import anthropic

class Item(BaseModel):
    description: str
    amount: float | None

class Receipt(BaseModel):
    merchant: str | None
    date: str | None
    items: list[Item]
    total: float | None

def encode(path, long_edge=1024):
    img = Image.open(path)
    img.thumbnail((long_edge, long_edge))         # resize BEFORE paying tokens
    buf = io.BytesIO(); img.convert("RGB").save(buf, "JPEG", quality=90)
    return base64.b64encode(buf.getvalue()).decode()

PROMPT = ("Extract this receipt into JSON with keys merchant, date (ISO), "
          "items (list of {description, amount}), total. Use null for "
          "anything unreadable. Output ONLY the JSON object.")

client = anthropic.Anthropic()
def extract(path):
    msgs = [{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64",
         "media_type": "image/jpeg", "data": encode(path)}},
        {"type": "text", "text": PROMPT}]}]
    for attempt in range(2):                      # one repair round
        resp = client.messages.create(model="claude-sonnet-4-5",
                                      max_tokens=1024, messages=msgs)
        text = resp.content[0].text.strip()
        if not text.startswith("{"):              # strip any code fence
            text = text[text.find("{"): text.rfind("}") + 1]
        try:
            r = Receipt.model_validate_json(text)
            print("usage:", resp.usage)
            return r
        except (ValidationError, json.JSONDecodeError) as e:
            msgs += [{"role": "assistant", "content": text},
                     {"role": "user", "content": f"Invalid: {e}. Return corrected JSON only."}]
    raise RuntimeError("extraction failed after repair")

r = extract("receipt1.jpg")
if r.total is not None and all(i.amount is not None for i in r.items):
    delta = abs(sum(i.amount for i in r.items) - r.total)
    print("arithmetic check:", "PASS" if delta < 0.01 else f"MISMATCH by {delta:.2f}")
print(r.model_dump_json(indent=2))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

PDF pipeline: route by document type

20 min

Take two PDFs: one text-native (export any doc to PDF) and one scanned-style (print a page, photograph it, or find a scan).
Create pdf_router.py: use pypdf to attempt text extraction; if a page yields more than ~200 characters of real text, treat it as text-native and use the extracted text directly (cost: zero tokens of vision).
Otherwise render the page to an image (pdf2image or a screenshot) and send it down yesterday's VLM path.
For the text-native PDF, print the first 500 extracted characters and confirm reading order is sane (multi-column PDFs often are not — note it if you see it).
Write a comment block: the routing rule, cost per page on each branch (estimate from the usage fields), and which branch your capstone corpus needs.

🐍 python — editable, runs in your browser

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

The chart interrogation

20 min

Take a screenshot of any real bar or line chart (a dashboard, a news graphic, one of your own matplotlib plots from Day 67). Send it to the VLM three ways and compare: (1) "Describe this chart" (free-form); (2) "Extract the data as JSON: series, x labels, approximate y values, and set a confidence field low/medium/high per value"; (3) same as 2, but first tell it the true values for two points you know, then ask for the rest.

Goal: produce a short writeup — where was it accurate, where did it guess, did the confidence field correlate with actual error, and did anchoring with two known values help? End with your personal rule for when chart-reading by VLM is shippable and when you must demand the underlying data.

Hints: the failure you are hunting is CONFIDENT interpolation — plausible values with no pixel evidence. The arithmetic-check mindset from guided applies: what internal consistency can you verify?

Ship before you stop

Receipts-to-ledger extractor with a measured accuracy number

Build extractor/ in your practice repo: a CLI that takes a folder of receipt/invoice images and emits ledger.jsonl (one validated Receipt per line) plus report.md. Hand-label a small golden sample first: for 6 documents, write the true merchant, date, total, and item count in labels.json. The report must state field-level accuracy against those labels (exact match for merchant/date, ±0.01 for totals), the arithmetic-check pass rate, and cost per document from usage data. Failures stay in the ledger flagged "needs_review": true — never silently dropped. This artifact previews Day 134: you just built your first golden set without being told to.

Rubric — check what you completed (0/6)

Runs on a folder and emits valid JSONL — schema-validated, repair loop includedSix-document golden label file exists and was written BY HAND before running the extractorReport states per-field accuracy against labels, not a vibes summaryArithmetic cross-check implemented; mismatches flagged needs_review, not droppedCost per document computed from real usage fieldsCommitted with a README explaining the OCR-vs-VLM routing choice

Common mistakes & misconceptions

Sending 4000px images by default. Providers downscale anyway; you pay tokens for pixels the model never sees. Resize to the smallest size that keeps the relevant text legible — or crop the region.
Free-form extraction. "Summarize this invoice" invites hallucinated fields; schema + validation + repair (Day 110) is the only production pattern.
Trusting extracted numbers without cross-checks. Line-items-sum-to-total and date-in-plausible-range catch confident misreads that schemas cannot.
Using the VLM on text-native PDFs. pypdf extraction is free and character-exact; burn vision tokens only on scans and layout-heavy pages.
Reading precise values off charts. VLMs interpolate plausibly; demand approximate values with confidence, or get the source data.
No labeled sample. Without even 6 hand-labeled documents you have no accuracy number, and "it looked right on two receipts" is how demos die in week two.

Knowledge check

Q1. Why does resizing an image client-side before sending it to a VLM usually save money without hurting accuracy?

Q2. A batch of vendor PDFs extracts perfectly with pypdf. The right pipeline is…

Q3. Your extractor reads a smudged total as "$142.50" with full confidence, but the line items sum to $124.50. What caught this, and what does it teach?

Go deeper — curated resources

docsClaude Docs — Vision (limits, token costs, formats) ↗25 min repoOpenAI Cookbook — multimodal examples ↗20 min docsLlamaIndex Docs — document loading & multimodal ↗20 min

If you have a third hour

Multimodal RAG — Two mainstream designs: extract-then-index (today's pipeline feeding Day 115's index) vs multimodal embeddings that embed images directly. Extract-then-index wins for text-heavy docs; direct embeddings for photos/diagrams.

Done means

Receipt extraction runs with validation + repair; arithmetic check implemented
Resize experiment done; token-cost knee identified
PDF router distinguishes text-native from needs-VLM pages
Extractor project committed with hand-labeled sample and measured accuracy
Quiz ≥ 2/3

How this connects

← Back: This is Day 110's structured-output discipline pointed at pixels, priced by Day 96's token math — patches are just more tokens for Day 94's attention to read. The ingest thinking extends Day 113-114's pipeline.

Forward →: Tomorrow adds ears and a voice (Day 131). Your hand-labeled 6-document sample is a miniature of the golden sets you formalize on Day 134, and document ingest quality feeds directly into your capstone's Day 136 RAG evaluation.