Training optimizes model parameters on datasets; inference serves predictions under latency and availability SLOs. Confusing the two — or ignoring serving constraints while designing training — is a classic interview miss.

• Pin model artifact digests in serving; never "latest" in production
• Batch inference for offline jobs; stream for online paths
• Quantization, distillation, and caching are inference-side cost levers
• Training-serving skew is its own failure class — keep feature parity

Data → Train → Eval → Register model
                           ↓
                    Package (image/weights)
                           ↓
              Shadow → Canary → Full serve
                           ↓
                    Monitor → Retrain trigger

A feature store centralizes feature definitions, materialization, and serving so training and inference share consistent logic. Interviews test whether you understand point-in-time correctness and online/offline splits.

1. Define features once; generate training rows with point-in-time joins
2. Serve the same feature values online that training would have seen
3. Version feature logic; breaking changes need model retrain
4. Monitor freshness and null rates per feature

Events / batch tables
         │
   Feature pipelines
      /        \
Offline store   Online store (Redis/Dynamo)
 (train join)        │
      │              ▼
   Train set      Model server reads features
      │              │
      └── same definitions ──┘

Training–serving skew is any mismatch between features or preprocessing at train time versus serve time. It silently destroys model quality and is a favorite ML systems interview probe.

• Log serving feature vectors and compare distributions to training
• Shadow compute offline features for a sample of live traffic
• Integration tests: same input → same features in both paths
• Treat schema evolution as a coordinated deploy across pipelines and models

Drift is distribution shift in inputs, features, or predictions over time. Concept drift changes the label relationship; data drift changes the input mix. Production ML interviews expect detection, triage, and retrain triggers.

1. Establish baselines from a stable production window
2. Alert on sustained drift, not single-day noise
3. Separate traffic-mix changes (new market) from true concept drift
4. Automate retrain only when labels and pipelines are trustworthy

Live traffic → log features/preds
                ↓
         Drift detectors
                ↓
        ┌───────┴───────┐
     Data issue      Model issue
     (fix ETL)      (retrain/canary)
                ↓
           Postmortem + thresholds

Offline pipelines process batches for training, backfills, and reports. Online pipelines serve low-latency features and predictions. Designs that ignore consistency between them fail at scale.

• Backfill story: how you recompute features after a bug
• Exactly-once vs at-least-once: know your consumer idempotency
• Late events: watermark policies for streaming features
• Cost: offline can recompute cheaply; online must be lean

     Events
     /    \
 Batch     Stream
   │         │
 Offline   Speed features
 features      │
   \         /
    Serving merge
         │
      Prediction

Serving patterns include synchronous APIs, batch scorers, streaming, and LLM gateways with queues. Choose based on SLA, fan-out, and whether GPUs are involved.

• Autoscale on queue depth and GPU utilization, not just CPU
• Canary by traffic % with automatic rollback on error/latency
• Isolate noisy neighbors: separate pools for huge prompts vs tiny classifiers
• Cache embeddings and frequent completions where safe