Case study · Architect · 35 min

Offline + Online LLM Eval Harness

Create a continuous evaluation system that gates prompt and model changes with golden sets, LLM-as-judge, and online metrics.

evalsCIquality

Interview opener

Design the eval harness that gates production prompt/model changes.

1.LLM-as-judge disagrees with humans — now what?
2.How large a golden set for the first gate?
3.Online vs offline metrics — which blocks the deploy?

Strong signals

Human-calibrated judges
Slice metrics by failure class
CI gate + canary online metrics

Hit 0/3

Weak signals (avoid)

Only thumbs-up in production
Uncalibrated LLM judge as sole truth

Requirements

Functional: golden Q&A sets; rubric judges; regression CI gate; online shadow eval
Non-functional: full suite < 20 min; deterministic seeds; versioned datasets
Coverage: faithfulness, relevance, toxicity, tool-correctness, latency/cost
Scale: 5 products × 2,000 cases; 10 PR runs/day

Back-of-envelope Estimation

Metric	Calculation	Result
Cases/run	2,000	2,000
Tokens/case	3K in + 500 out + 1K judge	4.5K
Tokens/run	2K × 4.5K	9M
Cost/run @ $2/1M	9M × $2/1M	~$18
CI cost/day	10 × $18	~$180
Judge disagreement audit	5% human review	100 cases/run

API Design

Endpoint	Method	Description
POST /v1/evals/runs	POST	{ suiteId, candidateConfig } → run_id
GET /v1/evals/runs/{id}	GET	Scores, diffs vs baseline
POST /v1/datasets	POST	Upload versioned golden JSONL
POST /v1/online/shadow	POST	Log production sample for offline replay

Data Model

Entity	Key fields	Notes
datasets	dataset_id, version, hash, schema	Immutable versions
cases	case_id, input, expected, rubrics	Supports multi-turn
runs	run_id, config_hash, scores_json, cost	Compare to baseline run
judgments	judgment_id, case_id, judge_model, score	Store rationales

High-level Design

CI gate flow

PR → CI → Load dataset@v → Run candidate pipeline
                 → Metrics (exact / rubric / LLM judge)
                 → Diff vs main baseline
                 → Pass if Δfaithfulness ≥ −1pt AND toxicity not↑
                 → Publish report artifact

Prod → sample 1% → shadow store → nightly replay vs new prompts

Deep Dive: LLM-as-Judge Bias

Use pairwise judgments with position swap to reduce order bias
Calibrate judge against 200 human-labeled cases quarterly
Separate reference-free (faithfulness) from reference-based (exact slot fill)
Never let the candidate model judge itself

Bottlenecks at Scale

Judge API rate limits — shard cases; cache identical prompts
Flaky nondeterminism — temperature 0 + seed; retry on schema fail
Suite runtime — parallel workers; smoke suite on every PR, full nightly

Failure Modes & Monitoring

SLO	Target	Alert
CI suite runtime	< 20 min	Warn if > 30
Judge–human kappa	> 0.6	Quarterly review
False gate fails	< 5%	Track engineer overrides
Online quality drift	< 3pt weekly	Page product owner

AI & FDE Interview Room