Case study · Architect · 30 min

Cost-Optimized LLM Inference Stack

Design a multi-model inference platform that routes traffic across sizes, batches requests, and hits cost and latency SLOs.

inferenceGPUrouting

Interview opener

Cut generation spend ~60% without tanking quality.

1.Which traffic should never touch the frontier model?
2.How do you detect silent quality regressions?
3.Cache key design under multi-tenant ACL?

Strong signals

Routing + cache + smaller models
Eval before/after on same golden set
Cost and quality on one dashboard

Hit 0/3

Weak signals (avoid)

Only 'use cheaper model'
No regression detection

Requirements

Functional: OpenAI-compatible API; model routing; streaming; tenant quotas
Non-functional: chat p95 TTFT < 800ms; availability 99.9%; unit cost −40% YoY
Scale: 120 peak QPS; avg 1.5K in / 400 out tokens
Constraints: mix of hosted API + self-hosted 70B/8B GPUs

Back-of-envelope Estimation

Metric	Calculation	Result
Tokens/sec peak	120 × (1500+400)	~228K tok/s
If all 70B	need ~40 H100s (rough)	high $
Route 70% to 8B	effective GPU need ~0.4×	~16 H100-eq
API fallback 10%	12 QPS × 1.9K × 86400	~2B tokens/day
API cost @ $3/1M	2B × $3/1M	~$6K/day worst
Target mix cost	self-host heavy	~$1.2K/day

API Design

Endpoint	Method	Description
POST /v1/chat/completions	POST	Compatible schema; router picks model
GET /v1/models	GET	Lists logical models (quality tiers)
POST /v1/admin/routing	POST	Update rules: intent → model

Data Model

Entity	Key fields	Notes
routing_rules	rule_id, match, model_id, weight	Versioned config
usage	tenant_id, model_id, tokens_in/out, ts	Billing + quotas
quality_labels	request_id, score	From eval/sample

High-level Design

Router + serving

Client → API GW → Router (complexity classifier)
                      ├─► cache hit? return
                      ├─► small model pool (8B, continuous batch)
                      ├─► large model pool (70B)
                      └─► hosted API fallback (circuit breaker)

Telemetry → cost dashboards + auto scale GPU replicas

Deep Dive: Routing Without Quality Loss

Train a tiny classifier on (prompt features → needed model tier) using eval labels
Cascade: try 8B; if confidence low or self-check fails, escalate to 70B
Speculative decoding: 8B drafts, 70B verifies for hard prompts
Quantization (AWQ/FP8) with eval gate before prod

Bottlenecks at Scale

KV-cache memory for long contexts — cap and summarize
Head-of-line blocking — separate latency classes / priority queues
GPU fragmentation — continuous batching (vLLM-style) essential

Failure Modes & Monitoring

SLO	Target	Alert
TTFT p95	< 800ms	Page if > 1.5s
$/1M tokens	budget band	Warn on +20%
Fallback rate	< 15%	Warn if > 25%
Quality vs golden	≥ baseline−1pt	Block deploy

AI & FDE Interview Room