Case study · FDE · 25 min

Defining AI Pilot Success Metrics

Choose measurable pilot KPIs that balance model quality, user value, and business ROI — and know when to stop.

FDEmetricspilot

Interview opener

Define pilot success that executives and users both accept.

1.Leading vs lagging indicators?
2.What is an automatic kill?
3.How do vanity metrics mislead here?

Strong signals

Outcome metrics tied to persona pain
Pre-agreed kill criteria
Instrumentation plan before build

Hit 0/3

Weak signals (avoid)

Only 'number of queries'
Success redefined after the fact

Requirements

Pilot: 8 weeks, 150 users, internal knowledge assistant
Need leading + lagging indicators; explicit expand/kill thresholds
Avoid vanity metrics (raw query count without outcomes)
Report weekly to buyer in one page

Back-of-envelope Estimation

Metric	Calculation	Result
Users	150	150
Target WAU	60% = 90	engagement
Minutes saved/query	hypothesis 8 min	survey + diary
Queries/user/week	4	360 queries/week
Value/week	360 × 8/60 × $75/hr	~$3,600
LLM cost/week	~$50	clear ROI if value real

Metrics Collection API

Endpoint	Method	Description
POST /v1/events	POST	query, citation_click, copy, thumbs, escalate
POST /v1/surveys	POST	CSAT / time-saved responses
GET /v1/pilot/report	GET	Weekly KPI snapshot JSON

Data Model

Entity	Key fields	Notes
kpis	name, target, actual, week	Time series
cohorts	user_id, trained, active	Funnel
outcomes	ticket_deflect, time_saved	Business

High-level Design

Metric layers

Quality: groundedness, citation precision (offline+online sample)
UX:      WAU, thumbs-up rate, p95 latency
Value:   time saved, ticket deflection, NPS
Decision: expand if quality≥85% AND WAU≥50% AND value>$X
          kill if quality<75% after week 4 OR WAU<25%

Deep Dive: Goodhart's Law

Don't reward query volume — reward successful task completion
Pair automatic metrics with 10 user interviews mid-pilot
Pre-register kill criteria with the buyer to avoid moving goalposts
Separate model quality fails from change-management fails

Bottlenecks at Scale

Sparse survey responses — in-product micro-survey after citation click
Confounded seasonality — compare to prior period baselines
Champion bias — sample end users, not only power users

Failure Modes & Monitoring

KPI	Target	Action
Groundedness	≥ 85%	Pause expand; fix retrieval
WAU	≥ 50%	Training + UX review
CSAT	≥ 4.0	Interview detractors
ROI proxy	> 5× cost	Buyer go/no-go

AI & FDE Interview Room