Case study · Architect · 40 min
LLM-Augmented Product Recommendations
Combine classical retrieval with LLM re-ranking and natural-language preference understanding for product recommendations.
recommendationsembeddingsfeature store
Interview opener
LLM-assisted product recommendations with a classic feature store.
- 1.Where does the LLM add value vs collaborative filtering?
- 2.Training-serving skew on features?
- 3.How do you A/B without wrecking latency SLO?
Strong signals
Hit 0/3
Weak signals (avoid)
- Replace the entire recsys with an LLM
- Ignore feature store paths
Requirements
- Functional: personalized homepage + "explain why"; NL preference updates
- Non-functional: homepage p95 < 200ms; LLM path optional async enrichment
- Metrics: NDCG@10, CTR, revenue/session; no regress vs collaborative filter baseline
- Scale: 2M DAU; catalog 5M SKUs; 500 QPS peak homepage
Back-of-envelope Estimation
| Metric | Calculation | Result |
|---|---|---|
| Homepage QPS peak | given | 500/sec |
| Candidate gen | ANN over user+item emb | ~5ms |
| LLM rerank share | 5% of sessions (explore) | 25/sec |
| Rerank tokens | 25/sec × 2K × 86,400 | ~4.3B/day |
| Rerank cost @ $0.5/1M | 4.3B × $0.5/1M | ~$2,150/day |
| Item embeddings | 5M × 768 × 4 B | ~15 GB |
Keep LLM off the hot path
Serve classical candidates in <200ms; use LLM for explanations and offline/nearline rerank batches.
API Design
| Endpoint | Method | Description |
|---|---|---|
| GET /v1/recs/home | GET | Authenticated; returns ranked SKUs + reason codes |
| POST /v1/prefs/nl | POST | Natural language preference → structured tags |
| POST /v1/recs/explain | POST | { userId, skuId } → NL explanation |
Data Model
| Entity | Key fields | Notes |
|---|---|---|
| user_features | user_id, emb, affinity_tags, updated_at | Feature store online KV |
| item_features | sku_id, emb, category, price, attrs | Refreshed on catalog change |
| impressions | event_id, user_id, sku_id, position, clicked | Training labels |
High-level Design
Homepage: Client → Recs API → Feature store (user emb)
→ ANN candidates (200)
→ Lightweight ranker (GBDT) → top 20
→ (async) LLM explainer cache
NL prefs: text → LLM extract tags → write feature store
Offline: impressions → train two-tower + ranker; eval NDCGDeep Dive: Eval Without Gaming CTR
- Hold out users temporally; report NDCG@10 and revenue, not just CTR
- Guardrail: diversity and brand-safety filters before LLM rewrite
- LLM explanations must not invent attributes — ground in item features
Bottlenecks at Scale
- ANN recall drop on catalog inserts — hourly incremental index builds
- Feature store hot keys for celebs — cache user emb at edge
- LLM cost — quota explore traffic; distill explanations to templates
Failure Modes & Monitoring
| SLO | Target | Alert |
|---|---|---|
| Homepage p95 | < 200ms | Page if > 350ms |
| CTR vs baseline | ≥ −1% relative | Warn on A/B |
| Empty recs rate | < 0.01% | Page |
| Explain groundedness | > 95% | Weekly eval |