Case study · FDE · 30 min

AI Feature Rollout Incident Response

Handle a production incident where a new prompt/model rollout spikes hallucinations and customer escalations.

FDEincidentrollback

Interview opener

Pilot leaks a confidential HR document on day 2.

1.Shut down entirely or contain?
2.What do you say to legal vs users in the first hour?
3.Permanent controls to prevent recurrence?

Strong signals

Containment + blast-radius audit
Root cause as ACL/control failure
Trust rebuild with verification

Hit 0/3

Weak signals (avoid)

Blame the model only
Quietly patch without communication

Requirements

Symptom: groundedness down 12 pts; tickets +3× after Tuesday deploy
Need: detect, mitigate, communicate, root-cause, prevent
Constraints: cannot take entire product offline; pilot still live
Roles: FDE on-call, customer champion, internal eng, support lead

Back-of-envelope Estimation

Metric	Calculation	Result
Affected queries/hour	2,000	2,000
Bad answer rate	18% (up from 6%)	+240 bad/hour
Support cost	240 × 8 min × $1.5/min	~$2.9K/hour
Rollback time	feature flag flip	~5 min
Full RCA window	48 hours	blameless doc

Control Plane APIs

Endpoint	Method	Description
POST /v1/flags/{name}	POST	Set rollout % or force previous prompt
POST /v1/kill-switch	POST	Disable generative answers; FAQ-only mode
GET /v1/incidents/{id}	GET	Status page payload for customer

Data Model

Entity	Key fields	Notes
configs	version, prompt_hash, model_id	Immutable
incidents	id, severity, start, mitigate_at	Timeline
eval_snapshots	config_version, scores	Compare pre/post

High-level Design

Incident flow

Detect (eval drift / ticket spike / groundedness)
   → Triage sev (customer impact)
   → Mitigate: flag to prior config OR kill-switch FAQ mode
   → Communicate: champion + status page ETA
   → Diagnose: diff prompt/model/index; replay failing cases
   → Fix + eval gate → canary 5% → 50% → 100%
   → RCA + action items (tests, alerts)

Deep Dive: Safe Rollouts

Every prompt/model change behind a flag with automatic eval compare
Canary on 5% traffic with auto-rollback if groundedness −3 pts
Keep last-known-good config hot for one-click revert
Customer comms template prepared before go-live

Bottlenecks at Scale

Slow detection without online eval sampling
No flag → code redeploy for rollback (unacceptable)
Finger-pointing delays mitigate — empower FDE to flip flags

Failure Modes & Monitoring

SLO	Target	Alert
Detect time	< 15 min	Automated eval sample
Mitigate time	< 30 min	Page if open > 30
Comms cadence	Every 30 min	Until resolved
Repeat incident	0 in 90d	Track action items

AI & FDE Interview Room