The Model Landscape & Local Inference
- Compare open-weights vs closed API models along cost, privacy, control, and capability axes
- Navigate the Hugging Face ecosystem: models, model cards, licenses, and the transformers library
- Run a small open model locally with the transformers pipeline and with Ollama
- Explain what quantization does at a preview level and why it makes laptops viable
- Read a model card and license critically enough to advise a customer
| Spaced-rep warm-up: due cards (decoding, scaling) | 10 min |
| ELI5 + tech read: the trade-off axes | 20 min |
| Guided: transformers local run + model card reading | 25 min |
| Guided: Ollama + local REST API | 15 min |
| Practice: three client memos | 20 min |
| Project: landscape worksheet + quiz + flashcards | 30 min |
Builds on: Day 102 — Decoding & sampling · Day 100 — The training pipeline (base vs instruct) · Day 87 — PyTorch & tensors
Walk into a good workshop and you see a tool wall: some tools you own outright — hang them wherever you like, modify them, no one can take them away — and for the huge specialized machines, you rent time at the machine shop across town. Neither is "better." Owning means control, privacy, and zero rental fees, but you maintain the tools and the biggest machines will not fit in your garage. Renting means instant access to the most powerful equipment on earth, but you pay per use, your workpiece leaves the building, and the shop can change its machines or prices.
Models are exactly this. Open-weights models (Llama, Mistral, Qwen, Gemma families) are tools you own: download the weights, run them on your hardware, fine-tune them, ship them in an air-gapped environment. Closed API models (Claude, GPT, Gemini) are the machine shop: frontier capability, zero ops burden, metered billing. A working AI engineer knows the whole wall — and today you hang your first owned tool by running a real open model on your own machine.
"Should we use an API or host our own model?" is a top-three customer question, and the honest answer is a trade-off matrix, not a slogan — data-privacy rules, cost curves at volume, latency floors, and capability requirements all pull differently. FDEs meet customers with air-gapped environments (Day 170) where APIs are simply unavailable. And you cannot fine-tune (Day 128), quantize (Day 129), or serve with vLLM (Day 155) without today's Hugging Face and local-inference fluency. This lesson turns "I've heard of Llama" into "I've run it, read its license, and know when to recommend it."
Guided practice
Run an open model with transformers
25 min- Install:
pip install transformers torch(in your venv, Day 17 habits). - Paste the starter. It loads a deliberately small instruct model (Qwen2.5-0.5B-Instruct, ~1GB download) so it runs on CPU.
- Note the two-artifact pattern: tokenizer AND model must match — this is Day 96 made operational.
- Run the same prompt at temperature 0.3 and 1.2 — confirm your Day-102 knobs are back in your hands.
- Time the generation with time.perf_counter and compute tokens/second. Write it down; you will compare against a hosted API on Day 107.
- Open the model's page on huggingface.co and read its model card: license, training summary, limitations. Note all three.
Ollama: local inference as a service
15 min- Install Ollama (see the repo README for your OS) and pull a small quantized model:
ollama run llama3.2:1b(or another ~1B model). - Chat with it in the terminal; ask the same context-window question and compare quality to the transformers run.
- Ollama also serves a local REST API on port 11434. Hit it with curl or httpx (Day 41 payoff): POST to
/api/generatewith a JSON body containing model and prompt fields, and watch the streamed responses. - Check the model file size on disk versus the parameter count. Compute bits per parameter and confirm you are looking at a ~4-bit quantized model.
- One-line note: what did Ollama abstract away compared to the transformers workflow?
On your own
The deployment-recommendation memo
20 minThree clients, three recommendations. Write 3-4 sentences each, naming specific trade-off axes:
(1) A hospital network wants clinical-note summarization; patient data must never leave their datacenter, and they have budget for GPUs. (2) A 5-person startup prototyping a legal-research assistant, currently at ~200 requests/day, needs maximum answer quality. (3) A SaaS company runs 40M short classification calls/month on a frontier API and is bleeding money; tasks are narrow and well-defined.
Hints: (1) is a control/privacy story; (2) is a capability + zero-ops story — do not make them buy GPUs; (3) is the cost-shape crossover — a small open or distilled model on owned/rented infra, foreshadowing Day 129's selection matrix. State what you would verify before committing (license, eval on their data).
Model landscape worksheet
Create model-landscape.md: a comparison table of five models you researched today on the Hub or provider docs — at least two open (different license types) and one closed API model — with columns: params/tier, license, context length, instruct or base, where it can run, one sentence on when you would pick it. Below the table, record your measured local tokens/sec for both runs and the three client recommendations from practice. Commit it. Day 129 (model selection) and Day 170 (customer environments) both extend this worksheet.
Common mistakes & misconceptions
- Treating "open" as one thing. Weights-available spans permissive Apache/MIT, conditional community licenses, and research-only releases — the license, not the download button, decides what a customer may ship.
- Recommending self-hosting to save money at low volume. The cost crossover needs sustained utilization; idle GPUs are the most expensive way to serve 200 requests/day.
- Loading a base model and judging it as a bad chatbot. Base models continue text (Day 100); grab the -Instruct variant for assistant behavior, and use the chat template.
- Ignoring the tokenizer pairing. Loading model A with tokenizer B produces garbage — the vocabulary mapping is part of the model contract (Day 96).
- Assuming quantization is free. 4-bit is remarkably good but not lossless; quality-sensitive workloads need evaluation at the target precision (Day 129).
- Forgetting that APIs update models under stable-sounding names while self-hosted weights are frozen — pin versions and re-run evals on provider updates (Day 145).
Q1. A customer with strict data-residency rules and steady high volume asks API vs self-hosted. The strongest self-hosting arguments are…
Q2. A 7B-parameter model file on disk is about 4.4GB. Its precision is roughly…
Q3. Why must you use apply_chat_template (or equivalent) with an instruct model?
Go deeper — curated resources
- Serve your transformers model behind FastAPI — Wrap today's pipeline in your Day-41 FastAPI skills: a /generate endpoint with a pydantic request model. This is a miniature of Day 155's serving lesson.
- A real open model generated text on your machine via transformers
- Ollama model pulled, chatted with, and hit via its REST API
- Model card and license read and summarized for both models
- Landscape worksheet committed with benchmarks and client memos
- Quiz ≥ 2/3
← Back: The chat template operationalizes Day 100's SFT format; the sampling flags you passed to generate() are Day 102's knobs; the tokenizer/model pairing is Day 96's lesson in production form.
Forward →: Day 106 crosses to the rented side of the wall — the Anthropic API. Day 128 fine-tunes an open model you can now run; Day 129 formalizes quantization and model selection; Day 155 serves open models properly with vLLM; Day 170 revisits all of this inside customer environments.