Day 170 · Cooking in someone else's kitchen

Customer Environments

You will be able to

Compare SaaS, customer-VPC, and on-prem/air-gapped deployment models across data residency, control, and support cost
Explain VPCs, private networking, and egress control well enough to plan a deployment into one
State what customer-managed keys (CMK) change about your architecture and operations
Design a debugging strategy for environments you cannot directly access
Produce an environment preflight checklist that catches blockers before install day

Today's ~120 minutes

Spaced-rep warm-up: due cards incl. Day 169	10 min
ELI5 + tech read: the three models	20 min
Guided: decision matrix + diagnostic bundle	40 min
Practice: preflight checklist	20 min
Project: assemble the environment kit	20 min
Quiz + flashcards	10 min

Builds on: Day 150 — Cloud fundamentals · Day 159 — Production security & compliance · Day 169 — Enterprise integration

The analogy

🧸 ELI5⚙ Tech

Cooking in your own kitchen, you know where every knife lives, the oven runs 10 degrees hot, and you can taste the sauce whenever you want. Now cook the same dinner in a stranger's kitchen — with the lights half off. Some drawers are locked. The host stands behind you saying "we don't allow open flames on Tuesdays." And you can't taste anything: you ask the host to taste it and describe the flavor back to you.

That is deploying into a customer's environment. Their cloud account or data center is their kitchen: their network rules decide what your app can reach, their compliance rules decide where data may sit, and their keys lock the pantry. The dimmed lights are the loss of observability — you often can't SSH in, can't tail logs live, sometimes can't reach the internet at all. The skill is cooking well anyway: package everything you need before you arrive, write down every step so the host can follow it, and build tools that let the host "taste and describe" — diagnostics they run and send back — because you will not be allowed to taste it yourself.

Why this matters on the job

The deals with the biggest budgets — banks, healthcare, government, defense — are precisely the ones that will not send their data to your SaaS. "Can you deploy in our VPC?" or "we are air-gapped" is where AI deals in regulated industries live or die, and the engineer who can say "yes, here is our deployment model and here is the support cost difference" is the FDE. It is also where careers get made at 2 a.m.: production is down in an environment you cannot see, and your diagnostic tooling is the only flashlight anyone has.

Guided practice

guided 1

Deployment-model decision matrix

18 min

Create docs/deployment-models.md in the capstone repo. Build a table: rows = SaaS / customer-VPC / on-prem-air-gapped; columns = where data rests, who holds keys, how updates ship, how you debug, LLM access path, relative support cost (1x / 3x / 10x).
Fill every cell for YOUR capstone specifically — e.g. the LLM row: SaaS = hosted API; VPC = provider endpoint via their egress allowlist OR cloud-native model service in-region; air-gapped = local open-weights model (Day 103) with the quality delta stated honestly.
Now profile three prospects: a 40-person startup, Meridian Insurance (Day 169), and a defense contractor. Pick a model for each and write ONE sentence of justification per choice.
Add a "what we would have to build" list for the air-gapped column — this is your honest engineering bill for saying yes to that deal.

guided 2

Build the diagnostic bundle

22 min

You cannot SSH into Meridian's VPC. Build the flashlight you will hand them instead.

Create tools/diagnostic_bundle.py from the starter. It must be safe to run in a customer environment: it reports env-var PRESENCE, never values, and redacts hostnames in output.
Run it locally against your capstone stack. Read the JSON it prints.
Add two checks the starter lacks: (a) reach the vector DB health endpoint; (b) confirm the configured model name matches an allowlist of expected values.
Test the failure paths: unset an env var and point one URL at a dead port; confirm the bundle REPORTS the failures instead of crashing on them.
Commit it. On Day 172 this script is your first move in two of the three debugging scenarios.

🐍 python — editable, runs in your browser

import json, os, platform, shutil, socket, sys, time
import urllib.request

# Diagnostic bundle: run inside the customer environment, send back the JSON.
# Rule 1: never print secret VALUES. Rule 2: never crash - a failed check is data.

REQUIRED_ENV = ["LLM_API_KEY", "VECTOR_DB_URL", "APP_ENV", "LOG_LEVEL"]
ENDPOINTS = {
    "app_health": os.environ.get("APP_URL", "http://localhost:8000") + "/health",
    "llm_egress": "https://api.anthropic.com",  # connectivity only, no auth call
}

def check_env():
    return {k: ("present" if os.environ.get(k) else "MISSING") for k in REQUIRED_ENV}

def check_endpoint(url, timeout=5):
    t0 = time.perf_counter()
    try:
        req = urllib.request.Request(url, method="GET")
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            code = resp.status
    except Exception as e:
        return {"ok": False, "error": type(e).__name__ + ": " + str(e)[:120]}
    ms = round((time.perf_counter() - t0) * 1000)
    return {"ok": code < 500, "status": code, "latency_ms": ms}

def check_disk():
    du = shutil.disk_usage("/")
    return {"free_gb": round(du.free / 1e9, 1), "pct_used": round(du.used / du.total * 100)}

def check_dns(host="api.anthropic.com"):
    try:
        socket.getaddrinfo(host, 443)
        return {"ok": True}
    except Exception as e:
        return {"ok": False, "error": str(e)[:120]}

bundle = {
    "collected_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    "python": sys.version.split()[0],
    "platform": platform.platform(),
    "env_vars": check_env(),
    "dns": check_dns(),
    "endpoints": {name: check_endpoint(url) for name, url in ENDPOINTS.items()},
    "disk": check_disk(),
}
print(json.dumps(bundle, indent=2))

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

On your own

The preflight checklist

20 min

Meridian's install is in two weeks. Write docs/preflight-checklist.md — the list their platform team completes BEFORE you show up.

Constraints: every item must be verifiable by the customer without you (phrase as commands or yes/no checks, not "ensure networking works"); cover at minimum egress allowlist entries (enumerate the exact hostnames your capstone calls), DNS, proxy/CA-bundle handling, compute sizing, service-account permissions for SharePoint and the IdP app registration (Day 169), secrets delivery, and who is in the room on install day with what access. Cap it at one page — a checklist nobody completes is worse than none.

Hints: read your own docker-compose and .env.example and list every external hostname; the CA-bundle item is the one everyone forgets in TLS-inspecting proxies.

Ship before you stop

Environment kit: models doc + bundle + preflight

Assemble today's three artifacts into a coherent "environment kit" in the capstone repo: docs/deployment-models.md (matrix + three prospect calls + air-gapped bill), tools/diagnostic_bundle.py (extended, failure-tested), and docs/preflight-checklist.md (one page, customer-executable). Add a short section to the README pointing support engineers at the kit: when a customer reports trouble, step 1 is "run the bundle, send the JSON." This kit is graded again inside Day 174's simulation, where the prospect demands VPC deployment.

Rubric — check what you completed (0/6)

Matrix filled for all three models including the LLM access path per modelThree prospect recommendations each carry a one-sentence justificationDiagnostic bundle runs green locally AND reports (not crashes on) two induced failuresBundle never emits a secret value — grep the output to prove itPreflight checklist enumerates the exact egress hostnames from your own configREADME updated with the support-engineer entry point

Common mistakes & misconceptions

Promising on-prem at SaaS support cost. Every deployment model to the right of SaaS multiplies versions in the wild and slows every debug loop — price and staff it, or lose money on every regulated deal.
Discovering egress rules on install day. Deny-by-default networks block your LLM API, your package pulls, and your telemetry; enumerate every outbound hostname in the preflight, not in the war room.
Forgetting the TLS-inspecting proxy. Customer proxies re-sign traffic with an internal CA; your Python/httpx stack must be pointed at their CA bundle or every HTTPS call fails with a "certificate verify" error that looks like a bug in your code.
Diagnostic tools that print secrets. A bundle that dumps env-var values will be (rightly) refused by the customer's security team — report presence and shape, never values.
Treating customer-managed keys as a checkbox. CMK means the customer can revoke access to their own data at any moment; your restore, migration, and support procedures must assume that power exists.
Debugging by asking the customer to "poke around." Untrained hands in a production environment create new incidents; give them ONE command (the bundle) and navigate the rest yourself on a screen-share.

Knowledge check

Q1. A bank says "your product may not send our data outside our AWS account." Which deployment model is the minimum fit?

Q2. Your app works everywhere except one customer, where every HTTPS call fails with a certificate-verification error. Most likely cause?

Q3. The first design rule of a diagnostic bundle script for customer environments is…

Go deeper — curated resources

docsAWS docs — What is Amazon VPC? ↗20 min courseAWS Skill Builder — networking fundamentals paths ↗30 min articleReflections on Palantir — what field deployment feels like ↗25 min

If you have a third hour

Air-gapped LLM serving — Sketch what replacing your hosted model with a local vLLM deployment (Day 155) would change: quality evals to re-run (Day 140), GPU sizing, and how model updates ship as files.

Done means

Deployment matrix committed with three justified prospect recommendations
Diagnostic bundle handles two induced failures without crashing; output secret-free
Preflight checklist fits one page and lists your real egress hostnames
Quiz ≥ 2/3

How this connects

← Back: Day 150 taught VPCs as YOUR infrastructure; today the same picture belongs to the customer and you are the guest. Day 142's redacted tracing and Day 159's security checklist are why anything is debuggable at all in there.

Forward →: Day 172 runs incident scenarios where the diagnostic bundle is your opening move. Day 173 prices the support-cost multipliers you tabulated today, and Day 174's prospect demands VPC deployment in writing.

Unlocks: D172 Production Debugging with Customers