Day 107 · The meter is running

LLM APIs II — Streaming, Retries & Cost

You will be able to

Stream responses token-by-token and explain why TTFT matters more than total latency for UX
Handle API failures with typed exceptions, honoring retry-after and exponential backoff
Explain rate-limit dimensions (RPM, input/output tokens per minute) and design within them
Do concrete cost math for a feature: per-request, per-month, and with prompt caching applied
Ship a robust reusable client wrapper that logs cost and latency on every call

Today's ~125 minutes

Spaced-rep warm-up: due cards (request anatomy, statelessness)	10 min
Tech read: streaming, failure taxonomy, cost math	20 min
Guided: blocking vs streaming instrumented	20 min
Guided: robust wrapper with retries + logging	25 min
Practice: price the feature + cache verification	25 min
Project: toolkit v2 + quiz + flashcards	25 min

Builds on: Day 106 — The request & usage fields · Day 40 — Async, timeouts & concurrency · Day 47 — Caching & backpressure

The analogy

🧸 ELI5⚙ Tech

Yesterday you rented the polymath. Today you notice three things every renter learns the hard way. First: if you wait for the polymath to finish a two-minute answer before showing anything, your user stares at a spinner and leaves. The fix is streaming — the polymath dictates and you relay word by word. What users feel is not total time but time-to-first-word.

Second: the phone line to the polymath sometimes drops, and sometimes the office says "too many calls, try again in 20 seconds." Amateur code crashes; professional code catches the specific failure, waits the polite amount (they literally tell you how long), and retries — a few times, with growing pauses, then gives up gracefully.

Third: the meter is always running, in both directions, and the biggest bills come from a detail beginners miss — you resend the same briefing memo and transcript every single turn. The fix is the provider's discount for repeated pages: prompt caching. Mark the stable prefix once and re-reads of it cost a fraction of full price. Streaming for feel, retries for reliability, caching for cost: the three habits that separate a demo from a product.

Why this matters on the job

The difference between a hackathon demo and a production feature is exactly today's material. Streaming is why chat products feel alive despite multi-second generations. Retry discipline is what keeps your service up during provider blips and rate-limit bursts — which WILL happen at the worst time. And cost engineering is a career skill: features get killed or funded on per-request unit economics, and the engineer who can say "this costs 0.6 cents per request, 0.2 with caching, here is the math" owns the room. The wrapper you ship today is the client used by your capstone and every project until then.

Guided practice

guided 1

Feel the difference: blocking vs streaming

20 min

Ask for a ~500-word explanation twice: once with messages.create (print only when complete), once with messages.stream (print as tokens arrive).
Instrument both with time.perf_counter: record TTFT (first visible character) and total time.
Typical finding: near-identical total time, radically different feel. Write the one-line UX lesson.
From the streaming run, call stream.get_final_message() and confirm usage fields are still available after streaming.
Compute tokens/second for the generation phase and compare with your Day-103 local number.

🐍 python — editable, runs in your browser

import time
import anthropic

client = anthropic.Anthropic()
PROMPT = 'Explain, in about 500 words, why streaming improves chat UX.'

# Blocking
t0 = time.perf_counter()
r = client.messages.create(
    model='claude-sonnet-5', max_tokens=1024,
    messages=[{'role': 'user', 'content': PROMPT}])
t_block = time.perf_counter() - t0
print('blocking: total', round(t_block, 2), 's (nothing visible until now)')

# Streaming
t0 = time.perf_counter()
ttft = None
n_chars = 0
with client.messages.stream(
    model='claude-sonnet-5', max_tokens=1024,
    messages=[{'role': 'user', 'content': PROMPT}],
) as stream:
    for text in stream.text_stream:
        if ttft is None:
            ttft = time.perf_counter() - t0
        n_chars += len(text)
        print(text, end='', flush=True)
    final = stream.get_final_message()
total = time.perf_counter() - t0
print()
print('streaming: TTFT', round(ttft, 2), 's | total', round(total, 2), 's')
print('output tokens:', final.usage.output_tokens,
      '| tok/s:', round(final.usage.output_tokens / (total - ttft), 1))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Build the robust wrapper

25 min

Create llm_client.py in your llm-toolkit repo with the starter below: one call() function wrapping messages.create with typed exception handling, jittered exponential backoff, and a per-call log line (latency, tokens, illustrative cost).
Test the happy path.
Simulate failure: temporarily set a bogus API key and confirm AuthenticationError is NOT retried (fail fast on your own bugs).
Read the backoff math: attempt n sleeps base * 2^n plus jitter, capped. Explain in one line why jitter prevents synchronized retry stampedes (Day 46 thundering herd).
Keep the JSONL cost log — Day 146's dashboards will read this format.

🐍 python — editable, runs in your browser

import json
import random
import time
import anthropic

client = anthropic.Anthropic()
IN_PRICE, OUT_PRICE = 3.00, 15.00   # illustrative USD per 1M tokens

def call(messages, system=None, model='claude-sonnet-5',
         max_tokens=1024, max_attempts=4):
    attempt = 0
    while True:
        try:
            t0 = time.perf_counter()
            kwargs = dict(model=model, max_tokens=max_tokens,
                          messages=messages)
            if system:
                kwargs['system'] = system
            resp = client.messages.create(**kwargs)
            dt = time.perf_counter() - t0
            cost = (resp.usage.input_tokens * IN_PRICE
                    + resp.usage.output_tokens * OUT_PRICE) / 1e6
            with open('llm_calls.jsonl', 'a') as f:
                f.write(json.dumps({
                    'ts': time.time(), 'model': model,
                    'latency_s': round(dt, 3),
                    'in_tok': resp.usage.input_tokens,
                    'out_tok': resp.usage.output_tokens,
                    'cost_usd_illustrative': round(cost, 6),
                    'stop_reason': resp.stop_reason,
                }) + '\n')
            return resp
        except anthropic.RateLimitError as e:
            attempt += 1
            if attempt >= max_attempts:
                raise
            retry_after = e.response.headers.get('retry-after')
            delay = float(retry_after) if retry_after else min(
                60, (2 ** attempt) + random.uniform(0, 1))
            print('rate limited; sleeping', round(delay, 1), 's')
            time.sleep(delay)
        except (anthropic.APIConnectionError,
                anthropic.InternalServerError) as e:
            attempt += 1
            if attempt >= max_attempts:
                raise
            delay = min(60, (2 ** attempt) + random.uniform(0, 1))
            print('transient error:', type(e).__name__,
                  '; retrying in', round(delay, 1), 's')
            time.sleep(delay)
        # BadRequestError / AuthenticationError propagate immediately:
        # those are OUR bugs — retrying cannot fix them.

if __name__ == '__main__':
    r = call([{'role': 'user', 'content': 'Say OK.'}])
    print(''.join(b.text for b in r.content if b.type == 'text'))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Price the feature, then cache it

25 min

A product manager specs an email-drafting assistant: 2,000-token system prompt (style guide + examples), average 250 tokens of user input, 350-token drafts, projected 50,000 drafts/month.

Deliver a costed proposal: (1) per-draft and monthly cost at illustrative prices of 3.00/M input and 15.00/M output — show the arithmetic; (2) the same with prompt caching on the system prompt (reads ~0.1× input price; assume warm cache), and the monthly saving; (3) verify empirically: add cache_control to your wrapper's system prompt as a content-block dict — {'type': 'text', 'text': SYSTEM, 'cache_control': {'type': 'ephemeral'}} — call twice, and show usage.cache_read_input_tokens > 0 on the second call; (4) one paragraph: which future choices (shorter prompts, smaller model, output caps) move the bill most, and why output tokens punch 5× above their weight.

Hints: label every number illustrative; the cached-prefix must come before any volatile content; if cache reads show zero, check the prefix-stability rule.

Ship before you stop

llm-toolkit v2: the production-ish client

Upgrade the toolkit: llm_client.py (wrapper with retries + JSONL logging), llm_chat.py switched to streaming output via the wrapper, and a new costs.py that reads llm_calls.jsonl and prints a session/day summary table (calls, tokens in/out, total illustrative cost, p50/p95 latency — Day 58's percentiles in the wild). README gains a "cost model" section with your email-assistant math, caching numbers, and a stated verify-current-pricing caveat. Commit and push. This client is imported by every subsequent day's code through the capstone.

Rubric — check what you completed (0/6)

Streaming chat shows tokens as they arrive and still logs final usageRate-limit path honors retry-after; transient errors back off with jitter; auth/bad-request fail fastEvery call appends a JSONL record with latency, tokens, cost, stop_reasoncosts.py reports totals plus p50/p95 latency from the logCache demo shows nonzero cache_read_input_tokens on the second callREADME cost model reproduces the practice math and labels prices illustrative

Common mistakes & misconceptions

Retrying everything. A 400/401 is your bug — retrying hammers the API and hides the defect. Only 429s, 5xx/overloaded, and network errors deserve retries.
Ignoring the retry-after header and inventing your own delay for 429s. The server told you exactly how long to wait; earlier retries fail anyway.
Backoff without jitter. Synchronized clients retry in lockstep and re-stampede the service — the Day-46 thundering herd, now billed per token.
Judging latency by total time. Users experience TTFT; a 6-second response that starts in 400ms feels fine, a 3-second blocking response feels broken.
Putting volatile content (timestamps, user IDs) at the TOP of the prompt. Caching is prefix-matching — one changed early byte invalidates everything after it.
Forgetting output tokens cost ~5× input at typical list prices. Verbose answers, not long prompts, often dominate the bill — cap and instruct for concision.

Knowledge check

Q1. Which failures should your client retry?

Q2. A 2,000-token cached system prompt is re-read on a request (cache reads ~0.1× the 3.00/M input price). The prefix now costs about…

Q3. Why can you hit rate limits while sending only 30 requests per minute?

Go deeper — curated resources

docsClaude Docs — Streaming Messages (SSE events & SDK helpers) ↗25 min docsClaude Docs — Rate limits (dimensions, headers, retry-after) ↗20 min docsClaude Docs — Models overview (current pricing to replace the illustrative numbers) ↗10 min repoAnthropic Python SDK — error types & retry configuration ↗15 min

If you have a third hour

Batch APIs for offline workloads — Providers offer async batch endpoints at ~50% discount for non-interactive jobs (evals, backfills). Keep this in your cost toolbox for Day 134's eval harness — eval suites are the perfect batch workload.

Done means

TTFT and total latency measured for both request modes
Wrapper retries correctly by failure class and fails fast on request bugs
Feature priced with and without caching; cache_read_input_tokens observed nonzero
costs.py summarizes the JSONL log with p50/p95 latency
Quiz ≥ 2/3

How this connects

← Back: The retry/backoff discipline is Day 46-47's distributed-systems hygiene applied to a paid dependency; percentile latency reporting comes from Day 58; the usage fields you now engineer around were introduced on Day 106.

Forward →: Day 110 adds structured outputs to this wrapper and Day 111 adds tools. Day 129 uses your cost log for model-selection decisions, Day 146 builds dashboards on the JSONL format, and Day 156 pushes caching and routing to production scale.

Unlocks: D109 Prompt Engineering II — Memos That Survive Contact · D110 Structured Outputs — Forms, Not Essays · D111 Tool Use & Function Calling · D129 Distillation, Quantization & Model Selection