Day 173 · Is the robot worth it?

ROI, Pricing & Cost Analysis

You will be able to

Build a full cost model for an AI feature: tokens, infra, and people — per query and per month
Quantify value honestly: time saved, ticket deflection, revenue effects, with an adoption ramp
Run a build-vs-buy comparison that includes total cost of ownership, not just sticker price
Design a pilot with success metrics agreed BEFORE it starts
Compress all of it into a one-page cost analysis an executive will actually read

Today's ~120 minutes

Spaced-rep warm-up: due cards from Days 169–172	10 min
ELI5 + tech read: three cost layers, three value currencies	20 min
Guided: cost model + value model as code	38 min
Practice: build-vs-buy memo	20 min
Project: the one-pager	22 min
Quiz + flashcards	10 min

Builds on: Day 107 — LLM API cost math · Day 156 — Caching, batching & cost engineering · Day 165 — Proposals & architecture docs

The analogy

🧸 ELI5⚙ Tech

A restaurant owner is offered a dishwashing robot: $2,000 a month. Worth it? A bad answer compares $2,000 to zero, because "we wash dishes for free now." A good answer prices the status quo: the dishwasher's wages, the plates broken, the Friday nights when dishes pile up and tables turn slower. Then it prices the robot honestly: $2,000 PLUS electricity, PLUS the plumber who installs it, PLUS the month where the staff still hand-wash because they don't trust it yet. Worth it means: total new cost vs total old cost, including the ramp where you pay both.

Every AI feature is the dishwashing robot. The token bill is just the sticker price — infra, integration work, and the humans who maintain and supervise it are the plumber and the electricity. The value side is never zero either, because the current way of doing things has a cost someone is already paying — it's just hidden in salaries and slow Fridays. Your job is to drag both totals into daylight, on one page, with the assumptions written down so people can argue with the INPUTS instead of with you.

Why this matters on the job

"What does this cost and what do we get?" is the question that decides renewals, expansions, and whether your pilot dies in procurement — and the engineer in the room is usually the only person who can answer it with real numbers. FDEs get pulled into pricing calls constantly: a cost model you can defend line-by-line beats a slide of vibes every time. It also protects YOU: features whose unit economics are underwater (Day 156's lesson) become your 2 a.m. problem when finance notices; better to be the one who noticed first.

Guided practice

guided 1

Cost model as code

20 min

Create cost_model.py from the starter — a transparent, assumption-driven model of your capstone at Meridian (4,000 employees).
Run it. Read the per-query cost and the monthly total at baseline assumptions.
Replace the placeholder token counts and prices with YOUR real numbers: pull mean input/output tokens from your Day 146 dashboard or Day 156 measurements, and your provider's current prices.
Run the three built-in scenarios: baseline, +semantic-cache at 30% hit rate, +routing 40% of queries to a small model. Record the monthly deltas.
Add a people-cost line: 0.25 engineer FTE for maintenance at a loaded cost you choose. Notice what it does to the total — and to which lever matters most.

🐍 python — editable, runs in your browser

# Transparent cost model - every number is an ASSUMPTION someone can argue with.

A = {
    "employees": 4000,
    "weekly_active_pct": 0.30,        # adoption at steady state
    "queries_per_user_week": 5,
    "in_tokens": 3200,                 # prompt + retrieved context (measure yours!)
    "out_tokens": 350,
    "price_in_per_mtok": 3.00,         # USD per million input tokens (update!)
    "price_out_per_mtok": 15.00,
    "embed_rerank_per_query": 0.0008,  # embedding + reranker cost per query
    "infra_monthly": 900.0,            # hosting, vector DB, observability
    "maint_fte": 0.25,                 # engineer fraction for evals/corpus/incidents
    "fte_loaded_annual": 180000.0,
}

def monthly_cost(a, cache_hit=0.0, small_route=0.0, small_discount=0.85):
    users = a["employees"] * a["weekly_active_pct"]
    q_month = users * a["queries_per_user_week"] * 4.33
    tok = (a["in_tokens"] * a["price_in_per_mtok"]
           + a["out_tokens"] * a["price_out_per_mtok"]) / 1e6
    per_q = tok + a["embed_rerank_per_query"]
    per_q *= (1 - cache_hit)                        # cache hits ~ free
    per_q *= (1 - small_route * small_discount)     # routed queries much cheaper
    llm = q_month * per_q
    people = a["maint_fte"] * a["fte_loaded_annual"] / 12
    total = llm + a["infra_monthly"] + people
    return {"queries": round(q_month), "per_query_usd": round(per_q, 4),
            "llm_usd": round(llm), "infra_usd": a["infra_monthly"],
            "people_usd": round(people), "total_usd": round(total)}

for name, kw in [("baseline", {}),
                 ("with 30% semantic cache", {"cache_hit": 0.30}),
                 ("cache + route 40% small", {"cache_hit": 0.30, "small_route": 0.40})]:
    r = monthly_cost(A, **kw)
    print(f"{name:28s} {r['queries']:>7} q/mo  " +
          f"llm ~{r['llm_usd']:>6}  people ~{r['people_usd']:>6}  TOTAL ~{r['total_usd']:>7}/mo")

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

The value side, with a straight face

18 min

Extend cost_model.py with the value model — same discipline, assumptions up top.

Time saved: active users × queries/week × minutes saved per query (estimate 4–8 min vs searching SharePoint or asking a colleague) × loaded hourly rate. Apply a productivity-conversion factor (claim 50%, defend it in a comment).
Deflection: Meridian's HR/IT helpdesks take ~1,400 policy-type tickets/month at ~USD 12 handled cost. Model deflection at 25% (pilot target) and 45% (steady state).
Apply an adoption ramp: months 1–3 at 25%/50%/75% of steady-state value while costs run at 100%.
Compute ROI and payback month. Print a 3-line summary.
Sensitivity: halve adoption AND minutes-saved. Is the project still above water? That worst-case number is the one you quote to a skeptical CFO — if it survives, you can't lose the argument on inputs.

On your own

Build-vs-buy memo

20 min

Meridian's IT director asks: "Why shouldn't we just buy DocsBot Enterprise at USD 6/user/month instead of your platform?"

Write a one-page memo comparing 3-year TCO of: (a) buying that SaaS tool, (b) deploying your capstone platform. Constraints: use your cost model's real numbers for (b); for (a) include license (be honest: 1,200 provisioned seats, not 4,000), integration and SSO setup, and the things sticker price hides — data leaves their tenant (Day 170 objection), limited permission-aware retrieval (Day 169), no custom evals. No straw-manning: state what the SaaS option does BETTER (time-to-value, zero maintenance). End with a recommendation and the single assumption that, if wrong, flips it.

Hints: the differentiator question decides more than the spreadsheet — is docs-QA plumbing or strategic for Meridian? TCO includes YOUR maintenance FTE line for option (b).

Ship before you stop

The cost-analysis one-pager

Write docs/cost-analysis.md for the capstone at Meridian — strictly one page. Sections: (1) Cost: per-query and monthly totals at baseline and optimized (cache + routing), the three layers visible; (2) Value: time-saved and deflection with the adoption ramp, stated as a range; (3) ROI + payback month, plus worst-case sensitivity in one sentence; (4) Top 3 assumptions anyone can challenge, each with its sensitivity; (5) Pilot proposal: 8 weeks, 200 users, success metrics pre-agreed (deflection ≥ 20%, weekly active ≥ 40% of pilot group, groundedness ≥ 90% on the golden set). Commit cost_model.py alongside so every number is reproducible. This one-pager is a required artifact in tomorrow's simulation and gets attacked by a CFO objection there.

Rubric — check what you completed (0/5)

Cost section shows all three layers; people-cost is present and non-trivialEvery value claim carries an assumption you could defend line-by-line, with a conversion discount on time savedROI and payback computed at baseline AND worst-case sensitivityPilot metrics are measurable and pre-agreed, not "customer satisfaction improves"Fits one page; cost_model.py committed and reproduces the numbers

Common mistakes & misconceptions

Comparing against zero. The status quo has a cost — salaries spent searching, tickets handled by humans, mistakes from stale answers. Price the current way of working first, or your robot always looks expensive.
Modeling tokens and forgetting people. Maintenance, evals, corpus upkeep, and incident response usually dwarf the token bill; a cost model without an FTE line is fiction that finance will eventually audit.
Claiming 100% of time saved as value. Three minutes saved forty times a day does not become two productive hours; apply and defend a conversion factor or lose credibility on every other number.
Running costs at day-one volume and value at steady state. Adoption ramps; costs don't. Model the crossover month honestly — that IS the payback story.
Pilots without pre-agreed metrics. If success criteria are negotiated after the data exists, the pilot proves whatever the loudest person wants; agree the bar in writing before week one.
Hiding assumptions to make the page look confident. Explicit, challengeable assumptions move arguments to inputs — where you win — instead of to your credibility, where nobody wins.

Knowledge check

Q1. Which cost layer most often dominates an enterprise AI feature's true monthly cost?

Q2. The honest way to model value during a rollout is…

Q3. A customer proposes "let's just run a pilot and see how it goes." Your required amendment:

Go deeper — curated resources

articleChip Huyen's blog — production LLM economics & architecture posts ↗25 min articlePreSales Collective — value engineering for technical sellers ↗15 min courseMade With ML — product & delivery lessons for ML systems ↗20 min

If you have a third hour

Unit economics over time — Extend cost_model.py to project 24 months with corpus growth (more context tokens), price declines (assume −30%/yr), and adoption growth. Which force wins?

Done means

cost_model.py runs with YOUR measured token numbers and all three layers
Worst-case sensitivity computed; you know if the project survives it
Build-vs-buy memo names what the buy option does better
One-pager committed at one page; quiz ≥ 2/3

How this connects

← Back: Day 107 taught the token meter, Day 156 taught the levers that bend it — today both became lines in a business case. Day 165's proposal template gets its "cost" section upgraded from estimate to model.

Forward →: Tomorrow's simulation (Day 174) fires a CFO objection directly at this one-pager. Day 177 re-runs the final capstone cost analysis as a quality gate, and Day 180's Demo Day presentation quotes your payback month.

Unlocks: D174 FDE Simulation II — Full Cycle