Case study · FDE · 40 min

Taking an AI PoC to Production

Plan the path from a flashy demo to a production AI feature with evals, security review, and a 6-week delivery timeline.

FDEdeliveryMVP

Interview opener

Three-week PoC that executives expect to become production.

1.IT security timeline is 6 weeks — your move?
2.Champion wants flash over foundations — push back how?
3.What artifacts remain if you leave day 21?

Strong signals

Risk-ordered backlog (access first)
Pilot metrics + kill criteria
Handoff: runbooks, owners, training

Hit 0/3

Weak signals (avoid)

Demo theater without data path
No success metrics

Requirements

Context: sales-led PoC chatbot demoed on sample PDFs; customer wants prod in 6 weeks
Must-have: SSO, ACL, audit logs, eval gate, support runbook
Success: 200 pilot users; CSAT ≥ 4.0; groundedness ≥ 85%; p95 < 5s
Constraints: customer VPC; no training on their data; change freeze week 6

Kill scope early

List explicit non-goals (agents, write-back, mobile). Map MoSCoW with the champion in week 0.

Back-of-envelope Estimation

Metric	Calculation	Result
Pilot queries/day	200 users × 5	1,000
LLM cost/day	1K × 3K tokens × $2/1M	~$6
Eng weeks	2 FDE + 1 SE × 6	18 eng-weeks
Security review buffer	1.5 weeks	parallelize docs early
Eval set size	150 golden Qs	~2 eng-days to label

API Design

Endpoint	Method	Description
POST /v1/ask	POST	Prod Q&A with SSO JWT
GET /v1/health	GET	Dependency checks for customer NOC
POST /v1/admin/reindex	POST	Trigger corpus refresh
GET /v1/metrics	GET	Prometheus scrape endpoint

Data Model

Entity	Key fields	Notes
pilot_users	user_id, cohort, trained_at	Track enablement
eval_cases	case_id, question, expected_docs	Customer-labeled
deployments	env, version, config_hash	Staging vs prod

High-level Design

6-week plan

W0: scope + data access + MoSCoW + threat model
W1: SSO + VPC deploy skeleton + sample ingest
W2: ACL retrieval + golden eval v1
W3: UX harden + logging/PII redaction
W4: security review + load test + runbooks
W5: pilot cohort on + feedback loop
W6: go-live checklist + hypercare + handoff docs

PoC demo ──► staging (customer VPC) ──► prod canary 10% ──► 100%

Deep Dive: PoC Debt to Pay Down

Replace hard-coded prompts with versioned configs
Add eval CI before any prompt tweak reaches prod
Wire real IdP groups — PoCs often fake ACLs
Document on-call ownership and rollback (previous model/prompt)

Bottlenecks at Scale

Data access delayed by legal — start DSARs week 0
Security questionnaire lag — answer in parallel with build
Champion vacation — identify backup decision maker

Failure Modes & Monitoring

SLO	Target	Alert
Pilot CSAT	≥ 4.0	Weekly review
Groundedness	≥ 85%	Block expand if < 80%
Sev-1 MTTR	< 60 min	Page FDE + customer IT
Rollback drill	Passed once	Before go-live

AI & FDE Interview Room