Case study · FDE · 40 min

On-Prem LLM Deployment in Customer VPC

Deploy a self-hosted LLM stack inside a regulated customer's VPC with air-gapped constraints and GPU sizing.

FDEon-premGPU

Interview opener

Customer mandates on-prem LLM; air-gapped constraints.

1.GPU capacity is half of what you need — options?
2.How do you update models under change control?
3.Who owns uptime after you leave?

Strong signals

Capacity plan + degrade modes
Ops ownership and runbooks
Model update + rollback procedure

Hit 0/3

Weak signals (avoid)

Assume infinite GPUs
No support handoff

Requirements

Air-gapped option: no egress; models delivered via secure USB/object drop
Latency: p95 < 3s for 2K context; HA across 2 AZs if GPUs allow
Compliance: SOC2 evidence pack; CMEK; audit logs to customer SIEM
Throughput: 15 QPS sustained; burst 40

Back-of-envelope Estimation

Metric	Calculation	Result
Tokens/sec needed	40 × 2,400	96K tok/s peak
70B FP8 throughput	~3–4K tok/s/H100	rough
GPUs for peak	96K / 3.5K	~28 H100
Steady 15 QPS	15/40 × 28	~11 H100
Plus embed + rerank	+2 A10	retrieval side
Power/cooling	28 × 700W	~20 kW

API Design

Endpoint	Method	Description
POST /v1/chat/completions	POST	Internal only; mTLS
POST /v1/embeddings	POST	Batch embed API
GET /v1/admin/license	GET	Offline license check

Data Model

Entity	Key fields	Notes
model_artifacts	name, version, checksum, path	Verified at load
nodes	gpu_id, model_loaded, health	Scheduler state
audit_events	ts, principal, action	SIEM forward

High-level Design

VPC layout

Customer VPC
  Private subnet: API pods (no egress) → GPU inference pool
                  Embed/rerank CPUs → Vector DB (disk encrypted)
  Management: bastion + image registry mirror (approved)
  Logs: Fluent Bit → customer SIEM
  Secrets: customer KMS-backed vault
Airgap update: signed model bundle → checksum verify → rolling restart

Deep Dive: Model Delivery & Rollback

Sign artifacts; verify checksum + signature before load
Blue/green model dirs; flip router weight after smoke eval
Keep N-1 weights on disk for 14-day rollback
Document GPU driver/CUDA matrix compatible with customer OS image

Bottlenecks at Scale

GPU procurement lead time — size for steady + burst queue
Single-AZ GPU scarcity — degrade to smaller model on overflow
Disk I/O loading 70B — stream weights; pin hot layers

Failure Modes & Monitoring

SLO	Target	Alert
p95 latency	< 3s	Page if > 5s
GPU util	60–85%	Warn if < 30% or > 95%
Queue depth	< 50	Autoscale / shed
Unsigned load attempts	0	Page security

AI & FDE Interview Room