Day 146 · The cockpit

Quality & Cost Dashboards

You will be able to

Choose the handful of metrics that actually belong on an AI-app dashboard
Compute latency percentiles (p50/p95) and cost-per-request from your logs and traces
Build a dashboard skeleton that turns the trace/log/eval streams into glanceable panels
Run a weekly review ritual and communicate quality to non-technical stakeholders

Today's ~120 minutes

Spaced-rep warm-up: tracing + drift cards (D142/145)	10 min
ELI5 + tech read: metric selection, percentiles, the ritual	20 min
Guided: aggregate metrics + render the dashboard	40 min
Practice: write the stakeholder weekly review	15 min
Project: build the capstone dashboard skeleton	25 min
Quiz + flashcards	10 min

Builds on: Day 142 — Tracing LLM applications · Day 143 — Logging, feedback & the data flywheel · Day 145 — Drift & continuous eval in prod

The analogy

🧸 ELI5⚙ Tech

A cockpit does not show the pilot everything the plane knows — it would be a wall of noise. It shows the six or seven instruments that change a decision: altitude, airspeed, heading, fuel, attitude, engine health. Everything else is one layer down, available when a warning light sends you looking. The art is choosing which few numbers earn a permanent place in front of the pilot's eyes.

A quality-and-cost dashboard is your cockpit for the docs-QA service. You already have the raw telemetry — traces (Day 142), logs and feedback (Day 143), continuous-eval scores (Day 145). A dashboard is not more data; it is the ruthless selection of the few numbers that would actually change what you do this week: is quality holding, is it fast enough, is it too expensive, are users unhappy, is anything erroring? Five panels a stakeholder can read in ten seconds beat fifty a nobody opens. And like a cockpit, it comes with a ritual: you don't glance once and forget — you scan the instruments every week, and the scan is what catches the slow leak before the passengers feel it.

Why this matters on the job

Two conversations decide whether an AI product survives, and both are dashboard conversations. The first is internal: "is it getting better or worse, and can we afford it?" — without cost-per-request and a quality trend, that turns into opinion. The second is the FDE's stakeholder update: an executive doesn't want your trace viewer, they want one screen that says quality is steady, p95 latency is under budget, cost per question is trending down, and here's the feedback rate. Being able to stand in front of a customer and narrate that screen — the "FDE muscle" the outline keeps naming — is often what renews the contract. Interviewers ask "what would you put on the dashboard for an LLM feature?"; a crisp, short answer signals you've actually run one.

Guided practice

guided 1

Aggregate logs and traces into panel metrics

22 min

Write obs/aggregate.py from the starter. It reads requests.jsonl (Day 143) and traces.jsonl (Day 142) and produces obs/dashboard_metrics.json: requests/day, p50/p95 latency, mean and total cost, error rate, refusal rate, feedback-down rate.
Implement the percentile function yourself (sort + index) — do not reach for a library, so you understand exactly what p95 means.
Join cost from the LLM spans (or from the request log's cost field) and confirm total cost = sum of per-request costs.
Run it: terminal: python obs/aggregate.py and eyeball the JSON — sanity-check that p95 ≥ p50 and error rate is in [0,1].
Fold in the Day-145 quality_timeseries.jsonl so the quality trend lands in the same metrics file.

🐍 python — editable, runs in your browser

# obs/aggregate.py — raw events -> dashboard metrics
import json, statistics

def load(path):
    try:
        return [json.loads(l) for l in open(path) if l.strip()]
    except FileNotFoundError:
        return []

def percentile(values, q):
    if not values:
        return None
    s = sorted(values)
    idx = min(len(s) - 1, int(q * len(s)))   # simple nearest-rank
    return s[idx]

def run():
    reqs = [e for e in load("requests.jsonl") if e.get("type") == "ask"]
    fb = [e for e in load("requests.jsonl") if e.get("type") == "feedback"]
    latencies = [r["latency_ms"] for r in reqs if "latency_ms" in r]
    costs = [r.get("cost_usd", 0.0) for r in reqs]
    errors = sum(1 for r in reqs if r.get("error"))
    refusals = sum(1 for r in reqs if "i can only answer" in r.get("answer", "").lower())
    downs = sum(1 for f in fb if f.get("rating") == "down")
    n = max(1, len(reqs))
    metrics = {
        "requests": len(reqs),
        "latency_p50_ms": percentile(latencies, 0.50),
        "latency_p95_ms": percentile(latencies, 0.95),
        "cost_mean_usd": round(statistics.mean(costs), 5) if costs else 0.0,
        "cost_total_usd": round(sum(costs), 4),
        "error_rate": round(errors / n, 3),
        "refusal_rate": round(refusals / n, 3),
        "feedback_down_rate": round(downs / n, 3),
    }
    quality = load("obs/quality_timeseries.jsonl")
    if quality:
        metrics["faithfulness_latest"] = quality[-1].get("mean_faithfulness")
    with open("obs/dashboard_metrics.json", "w") as f:
        json.dump(metrics, f, indent=2)
    print(json.dumps(metrics, indent=2))

if __name__ == "__main__":
    run()

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Render a five-panel dashboard page

18 min

Write obs/render_dashboard.py that reads dashboard_metrics.json and writes obs/dashboard.html — a self-contained page (inline CSS, no external calls) with five cards: Quality, Latency (p50/p95), Cost/request, Reliability (error + refusal), Volume.
Colour each card green/amber/red against the thresholds you set (e.g. p95 > 3000ms = amber, faithfulness < 0.80 = red). Thresholds live in one dict at the top.
Open the HTML in a browser and confirm a stakeholder could read it in ten seconds.
Add a one-line "as of <date>" and the total daily cost prominently — the number finance opens the page for.
Keep it dumb on purpose: the intelligence is in the metric choice and thresholds, not the chart library.

🐍 python — editable, runs in your browser

# obs/render_dashboard.py — metrics.json -> a glanceable HTML cockpit
import json, datetime

THRESH = {   # (amber, red) — direction noted per metric
    "faithfulness_latest": (0.85, 0.80),      # below is worse
    "latency_p95_ms": (2000, 3000),           # above is worse
    "feedback_down_rate": (0.05, 0.10),       # above is worse
    "error_rate": (0.01, 0.03),               # above is worse
}

def colour(key, val):
    if val is None or key not in THRESH:
        return "#888"
    amber, red = THRESH[key]
    worse_above = key != "faithfulness_latest"
    if worse_above:
        return "#c0392b" if val >= red else "#e67e22" if val >= amber else "#27ae60"
    return "#c0392b" if val <= red else "#e67e22" if val <= amber else "#27ae60"

def card(title, value, key=None):
    c = colour(key, value) if key else "#333"
    return (f'<div style="border:1px solid #ddd;border-radius:8px;padding:16px;'
            f'min-width:150px"><div style="font-size:13px;color:#666">{title}</div>'
            f'<div style="font-size:28px;font-weight:700;color:{c}">{value}</div></div>')

def run():
    m = json.load(open("obs/dashboard_metrics.json"))
    cards = "".join([
        card("Faithfulness", m.get("faithfulness_latest", "n/a"), "faithfulness_latest"),
        card("Latency p95 (ms)", m.get("latency_p95_ms", "n/a"), "latency_p95_ms"),
        card("Cost / request ($)", m.get("cost_mean_usd", 0)),
        card("Error rate", m.get("error_rate", 0), "error_rate"),
        card("Thumbs-down rate", m.get("feedback_down_rate", 0), "feedback_down_rate"),
        card("Requests today", m.get("requests", 0)),
        card("Total cost today ($)", m.get("cost_total_usd", 0)),
    ])
    html = (f"<!doctype html><meta charset=utf-8><title>Docs-QA cockpit</title>"
            f"<body style='font-family:system-ui;margin:24px'>"
            f"<h2>Docs-QA — quality & cost</h2>"
            f"<p style='color:#888'>as of {datetime.date.today()}</p>"
            f"<div style='display:flex;flex-wrap:wrap;gap:12px'>{cards}</div></body>")
    open("obs/dashboard.html", "w").write(html)
    print("wrote obs/dashboard.html")

if __name__ == "__main__":
    run()

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

Write the weekly review, for a stakeholder

15 min

Run your aggregator on the traffic you've generated this week and write the actual weekly-review note a non-technical stakeholder would receive: three to five sentences, no jargon, stating whether quality held, whether latency and cost are within budget, the feedback rate, and the single most important action for next week. Then, separately, list the three traces (by ID) you would pull to investigate the worst panel, and which of them you'd file as a flywheel case. The skill being graded is translation: turning five numbers into a paragraph an executive trusts.

Hints: lead with the answer ("Quality is steady; cost is up 12% and here's why"), not the methodology. If a number crossed a threshold, name the action, not just the number.

Ship before you stop

Build the capstone dashboard skeleton

Ship the cockpit for your docs-QA service. Commit obs/aggregate.py, obs/render_dashboard.py, and a generated dashboard.html, wired to your real logs and traces. The dashboard must show, at minimum: online faithfulness trend (Day 145), p50/p95 latency, cost per request and total daily cost, error rate, refusal rate, and feedback-down rate — each colour-coded against a documented threshold. Add obs/review.md: the weekly-review ritual written down (what to scan, the thresholds, what a crossing triggers, the three-sentence stakeholder summary template) plus this week's filled-in review. Make make dashboard (or a shell script) regenerate everything from raw logs in one command.

Rubric — check what you completed (0/6)

aggregate.py computes p50/p95 (hand-rolled percentile), mean/total cost, error and refusal rates from real logs/tracesdashboard.html renders five+ panels, colour-coded against documented thresholds, readable in ten secondsQuality trend from Day 145 appears alongside cost and latencyreview.md defines the weekly ritual and includes one completed stakeholder-facing summaryOne command regenerates the dashboard from raw logsThresholds are justified (why p95 > 3s is red, why faithfulness < 0.80 is red)

Common mistakes & misconceptions

Plotting everything you can measure. A fifty-panel dashboard is a wall nobody reads; pick the five numbers that change a decision.
Reporting average latency. The mean hides the slow tail users actually feel — report p50 and p95, and never average two percentiles together.
Omitting cost. Quality-only dashboards get the product killed by a surprise invoice; cost per request belongs next to quality, always.
Building the dashboard and skipping the ritual. Observability is the weekly scan, not the HTML; without the review the slow leak still strands you.
Showing engineers' internals to executives. Stakeholders need one glanceable screen and a three-sentence narrative, not your trace viewer.
Treating refusal rate as harmless. A refusal-rate spike is often retrieval failing silently before any error is logged — watch it as a leading indicator.

Knowledge check

Q1. Why report p95 latency instead of average latency?

Q2. Which set best belongs on an LLM-app cockpit?

Q3. A dashboard exists but quality still slips unnoticed for weeks. The most likely missing piece?

Go deeper — curated resources

docsGrafana — dashboards documentation ↗20 min docsPrometheus — overview (metrics model) ↗15 min articleChip Huyen — ML systems monitoring (metrics that matter) ↗20 min

If you have a third hour

Grafana + Prometheus for LLM apps ↗ — Sketch how each of your five panels would map to a Prometheus metric type (counter, gauge, histogram) — histograms are how real p95 is computed at scale.

Done means

aggregate.py produces p50/p95, cost, error/refusal rates from real logs/traces
dashboard.html renders colour-coded panels readable at a glance
review.md defines the weekly ritual with a completed stakeholder summary
One command regenerates the dashboard end to end
Quiz ≥ 2/3

How this connects

← Back: This turns Day 142's traces, Day 143's logs/feedback, and Day 145's quality time series into glanceable panels, and reuses Day 58's percentile intuition for the latency tail.

Forward →: Day 147 requires the dashboard populated for the observability gate; Day 157 graduates these panels into formal SLIs/SLOs with error budgets in Grafana/Prometheus, and Day 173 uses cost-per-request for the ROI one-pager.

Unlocks: D147 Observability Complete · D157 Monitoring & SLOs · D162 The FDE Role