Evals turn AI quality from opinion into a regression suite. Interviewers look for golden sets, automated graders, human review sampling, and gates that block bad prompt/model deploys.

1. Build a frozen golden set before optimizing prompts
2. Split online metrics (thumbs, CSAT) from offline regression
3. Version datasets; track which release was evaluated on which set
4. CI gate: fail deploy if core metrics drop beyond threshold

Change (prompt/model/index)
        ↓
 Offline suite (golden + safety)
        ↓
 Shadow / canary traffic
        ↓
 Online metrics + human sample
        ↓
 Promote or rollback

Offline evals are reproducible and cheap to iterate; online evals capture real user distribution and preference. Production AI design needs both, with clear promotion criteria between them.

• Interleave or A/B carefully; holdout users for measurement
• Use CUPED / variance reduction when effects are small
• Watch for Simpson's paradox across segments (locale, plan tier)
• Kill switches for experiments that spike toxicity or cost

Guardrails are layered controls: input filters, policy prompts, output classifiers, tool permissioning, and human escalation. Safety interviews reward defense-in-depth over a single system prompt ban list.

• Fail closed on high-severity categories; fail open carefully on low-severity with logging
• Separate safety evals from product quality evals
• Red-team continuously — attackers iterate on jailbreaks
• Document residual risk for legal/compliance stakeholders

User input
  → Input filters / rewrite
  → LLM (+ policy system prompt)
  → Output classifiers
  → Tool policy (if any)
  → Response or safe refusal

Every AI feature needs an explicit latency SLO and a unit-economics model. Interviews push you to allocate milliseconds and dollars across retrieval, models, and tools — then name the knobs.

1. Write a p50/p99 budget waterfall (auth + retrieve + LLM + post)
2. Estimate $/1k queries at expected QPS and cache hit rate
3. Set max tokens and max tool steps in config — not tribal knowledge
4. Prefer early exits: classify → retrieve → generate only if needed

Gateway/auth     50ms
Retrieve+rerank 200ms
LLM TTFT        400ms
LLM generate   1500ms
Post/validate    50ms
────────────────────
Total          2200ms  (300ms slack)

LLM observability extends classic metrics/logs/traces with prompts, retrieval sets, tool traces, token counts, and quality scores — under strict redaction. Without it, production incidents are un-debuggable.

• Trace id across gateway → retrieve → model → tools
• Sample raw prompts; never store secrets or full PII by default
• Dashboards: quality, cost, latency, safety hits — same pane of glass
• Alert on distribution shifts, not only hard errors

[trace_id]
 gateway → retrieve (chunk_ids)
        → llm (model, tokens, TTFT)
        → tool_x (args hash, latency)
        → llm
        → response (policy flags)
 feedback ←── user

AI incidents include quality cliffs, cost explosions, safety bypasses, and provider outages. Treat model/prompt/index changes like code deploys with instant rollback and clear severity.

1. Feature flags for prompt packs, models, and retrieval configs
2. Blue/green or alias-based index swaps
3. Runbooks with owners, dashboards, and customer comms templates
4. Post-incident: add an eval case that would have caught it