Day 131 · Ears and a voice

Multimodal II — Audio & Voice

You will be able to

Explain how Whisper-class STT works (audio → mel spectrogram → encoder-decoder transformer)
Transcribe real audio locally and compute word error rate against a reference by hand
Break a voice interaction into its latency budget (STT + LLM + TTS) and find the bottleneck
Explain endpointing and barge-in and why they make or break voice UX
Build a transcribe-and-summarize pipeline with speaker-aware structured output

Today's ~120 minutes

Spaced-rep warm-up: due cards (vision, quantization)	10 min
ELI5 + tech read; sketch the voice loop with budgets	20 min
Guided: local transcription + WER, latency budget lab	40 min
Practice: structured meeting minutes	20 min
Project: blueprint + transcribe CLI	20 min
Quiz + flashcards	10 min

Builds on: Day 95 — The Transformer (encoder-decoder) · Day 107 — Streaming & latency UX · Day 110 — Structured outputs

The analogy

🧸 ELI5⚙ Tech

Your polymath just got ears and a voice — and suddenly the rules of conversation apply. When you talk with a person, the rhythm is unforgiving: a reply that takes three seconds feels broken, because humans answer each other in well under one. So a voice assistant is a relay race run against a stopwatch: ears transcribe your words, the brain composes an answer, the mouth speaks it — and the whole relay must finish in about the time it takes you to say "um."

Two small skills separate a natural conversation from a walkie-talkie. Knowing when you have STOPPED talking (endpointing): pause too eagerly and it interrupts your mid-sentence breath; wait too long and every exchange gains an awkward second. And being interruptible (barge-in): when you start talking over its answer, a polite assistant shuts up immediately, exactly like a person would. Neither skill is about intelligence — both are about timing. That is the theme of today: voice is 20% transcription quality and 80% latency engineering.

Why this matters on the job

Voice is where AI meets the phone call — support lines, drive-through ordering, field-worker hands-free tools, accessibility interfaces — and it is a growing slice of FDE work because the engineering is unforgiving in ways text never is. A chatbot that takes 4 seconds is fine; a voice agent that takes 4 seconds is unusable, and the customer can hear exactly how bad it is in the first ten seconds of a demo. Knowing the latency anatomy — and that streaming everything (STT, LLM, TTS) is the only way to hit budget — is the difference between a wow demo and a dead one. Transcribe-and-summarize alone (meetings, calls, voicemails) is one of the most-requested enterprise features in existence.

Guided practice

guided 1

Transcribe locally and score WER by hand

25 min

Install: pip install openai-whisper (pulls PyTorch; also needs ffmpeg) — or faster-whisper if you prefer. Model base is enough.
Record ~30 seconds of yourself reading a paragraph you have as text (your phone's voice memo, exported as .m4a/.wav). That text is your reference.
Run the starter: it transcribes the audio, then computes WER against your reference with a from-scratch edit-distance implementation — no library, so you see that WER is Day 34's DP on words.
Read the alignment output: which words were substituted/dropped? Names and numbers fail first — note it.
Stress test: re-record the same paragraph with background noise (music, street) and with fast mumbling. Record all three WERs. Then transcribe 10 seconds of pure silence and observe what Whisper does with it.

🐍 python — editable, runs in your browser

import whisper

def wer(ref: str, hyp: str):
    r, h = ref.lower().split(), hyp.lower().split()
    # edit distance on WORDS — Day 34's DP table
    d = [[0] * (len(h) + 1) for _ in range(len(r) + 1)]
    for i in range(len(r) + 1): d[i][0] = i
    for j in range(len(h) + 1): d[0][j] = j
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            cost = 0 if r[i-1] == h[j-1] else 1
            d[i][j] = min(d[i-1][j] + 1,        # deletion
                          d[i][j-1] + 1,        # insertion
                          d[i-1][j-1] + cost)   # substitution/match
    return d[len(r)][len(h)] / max(1, len(r))

model = whisper.load_model("base")
result = model.transcribe("reading.m4a")
hyp = result["text"].strip()
ref = open("reference.txt").read().strip()
print("transcript:", hyp)
print(f"WER: {wer(ref, hyp):.1%}")
for seg in result["segments"]:
    print(f"[{seg['start']:6.1f}-{seg['end']:6.1f}] "
          f"(no_speech={seg['no_speech_prob']:.2f}) {seg['text']}")
# rerun on noisy.m4a, mumble.m4a, silence.m4a and compare WER +
# watch no_speech_prob — that field is your hallucination alarm.

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Latency budget: the voice-turn spreadsheet in code

15 min

Create voice_budget.py with the starter. It models a voice turn as stages with realistic latencies and compares three architectures: fully sequential, streamed STT, and fully streamed (overlapped LLM + sentence-streaming TTS).
Run it and read the "time to first audio" for each. Only the fully streamed design gets under 1 second.
Change one assumption at a time: a 1,500 ms LLM TTFT (big model), a 100 ms hosted STT, an on-device TTS at 80 ms. Which single change moves time-to-first-audio most? That is your bottleneck — write it in a comment.
Add a fourth architecture in code: cascade routing (Day 129) where a small fast model answers simple turns. Estimate its blended time-to-first-audio.

🐍 python — editable, runs in your browser

STAGES = {                       # ms, defensible 2026-ish defaults
    "endpoint_wait": 300,        # silence needed to decide "they stopped"
    "stt_final": 400,            # finalize transcript after end of speech
    "llm_ttft": 600,             # first token from the model
    "llm_first_sentence": 900,   # enough tokens to speak a sentence
    "tts_first_audio": 300,      # synthesis start-up
}

seq = (STAGES["endpoint_wait"] + STAGES["stt_final"]
       + STAGES["llm_first_sentence"] + STAGES["tts_first_audio"])

streamed_stt = (STAGES["endpoint_wait"] + 100      # transcript ~ready at stop
                + STAGES["llm_first_sentence"] + STAGES["tts_first_audio"])

fully_streamed = (STAGES["endpoint_wait"] + 100
                  + STAGES["llm_ttft"]             # TTS starts on sentence 1...
                  + 200)                           # ...overlapped synthesis

print(f"sequential:      {seq} ms to first audio")
print(f"streamed STT:    {streamed_stt} ms")
print(f"fully streamed:  {fully_streamed} ms   <- the only one under ~1.2s")
print("human comfort threshold: ~500-1000 ms")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Meeting minutes, structured

20 min

Build minutes.py: feed the transcript from guided exercise 1 (or any longer recording — a podcast clip you have rights to works) through an LLM with a Day 110 structured-output contract: {summary, decisions: [], action_items: [{owner, task, due}], open_questions: []} with nulls allowed for unknown owners/dates.

Constraints: the prompt must instruct the model to only include items actually said (no invented owners), timestamps from Whisper segments must be preserved on decisions, and the output must validate against a pydantic schema with one repair round.

Test with a trap: your recording should contain one vague statement ("someone should probably look at the logs") — does the model invent an owner? Write two sentences on the result.

Hints: hallucinated attribution is THE failure mode of meeting summarizers; the null-allowed schema plus explicit "use null if unstated" instruction is the countermeasure.

Ship before you stop

Voice-tutor architecture blueprint (paper) + transcribe-summarize CLI (code)

Two deliverables. (1) voice_notes.md: the architecture for a voice tutor (the far-future ambition of this program's platform) — a labeled pipeline diagram in text (capture → endpointing → streaming STT → LLM with conversation state → streaming TTS → playback with barge-in), the latency budget per stage with your numbers from guided 2, the two hardest engineering problems (you should now know they are endpointing and overlap orchestration, not "the AI"), and where a WER regression would visibly hurt. (2) transcribe.py: a CLI — python transcribe.py audio.m4a — emitting minutes.json per your practice schema plus a WER self-check mode when a reference file is supplied. Commit both.

Rubric — check what you completed (0/6)

Pipeline diagram names every stage including endpointing and barge-inLatency budget sums per architecture and identifies the bottleneck stageCLI transcribes a real file and emits schema-valid minutes.jsonWER mode computes against a reference using your own edit-distance codeHallucinated-attribution trap documented with the observed behaviorCommitted to the practice repo

Common mistakes & misconceptions

Treating voice as "chatbot + microphone". The hard parts are timing: endpointing, streaming overlap, barge-in. A perfect answer at 3 seconds loses to a good answer at 800 ms.
Sequential pipeline architecture. STT-then-LLM-then-TTS in series cannot hit human latency; every stage must stream and overlap.
Trusting Whisper on silence and noise. It hallucinates fluent text from nothing — check no_speech_prob and trim silence before transcribing.
Quoting WER without conditions. WER on clean read speech says nothing about accented, noisy, jargon-heavy calls — report WER per condition, like any eval slice (Day 80).
Letting the summarizer invent owners and dates for action items. Null-allowed schema + "only if stated" instruction + a trap test case.
Ignoring privacy: audio is often the most sensitive data a customer has. Local STT (Whisper on-prem) is frequently a requirement, not an optimization.

Knowledge check

Q1. A reference has 20 words; the transcript has 2 substitutions, 1 deletion, 1 insertion. WER?

Q2. Your voice agent feels laggy despite a fast LLM. Measurements: endpointing 300ms, STT 400ms after speech ends, LLM TTFT 500ms, TTS 300ms — sequential. Highest-leverage fix?

Q3. Whisper outputs a fluent sentence for a segment that was actually silence. What is this, and what flags it?

Go deeper — curated resources

repoopenai/whisper — repo, model card & usage ↗20 min paperWhisper paper — Robust Speech Recognition via Large-Scale Weak Supervision ↗30 min repoOpenAI Cookbook — audio examples ↗15 min

If you have a third hour

Speech-to-speech models — Native audio-in/audio-out models collapse the STT→LLM→TTS relay into one model, cutting latency and preserving tone/emotion — at the cost of the modular observability you get from a pipeline. The trade-off to watch.

Done means

Local transcription runs; WER computed with your own DP code across 3 conditions
Silence hallucination observed and its detection flag identified
Latency lab run; bottleneck stage named for your assumptions
Blueprint + CLI committed; minutes.json validates
Quiz ≥ 2/3

How this connects

← Back: Whisper is Day 95's encoder-decoder transformer with spectrograms for input; WER is Day 34's edit distance; the streaming-overlap obsession is Day 107's TTFT thinking, three stages deep.

Forward →: Tomorrow (Day 132) you learn why a voice channel is also an attack surface — spoken injection is still injection. Latency budgets return as p50/p95 engineering on Day 155, and the voice-tutor blueprint is a genuine future-product sketch.