Case study · Architect · 30 min

Semantic Cache for LLM Responses

Cut LLM spend with an exact-plus-semantic response cache that preserves freshness and per-user security boundaries.

cacheembeddingscost

Interview opener

Design a semantic cache for repeated RAG queries.

1.Near-duplicate malicious query — cache poisoning?
2.ACL: user A cached answer served to user B?
3.When is exact-match enough?

Strong signals

Cache keys include identity/ACL scope
Similarity threshold + TTL + invalidation on corpus change
Exact cache for FAQs first

Hit 0/3

Weak signals (avoid)

Global cache without tenancy
No invalidation story

Requirements

Functional: exact hash cache + near-duplicate semantic hits; TTL; tenant isolation
Non-functional: cache lookup p99 < 30ms; false semantic hit rate < 1% on eval
Target: 40% LLM call reduction on support FAQ traffic
Scale: 80 QPS; 10M cached embeddings

Back-of-envelope Estimation

Metric	Calculation	Result
LLM calls/day before	80 × 86,400	~6.9M
After 40% hit	0.6 × 6.9M	~4.1M
Saved tokens/day	2.8M × 2K	~5.6B
Saved $ @ $2/1M	5.6B × $2/1M	~$11.2K/day
Embed lookup cost	6.9M × 100 tok × $0.02/1M	~$14/day
Vector RAM	10M × 768 × 4 B	~30 GB

API Design

Endpoint	Method	Description
POST /v1/cache/lookup	POST	{ prompt, tenantId, aclKey } → hit\|miss + response
POST /v1/cache/put	POST	Store response with TTL and embedding
DELETE /v1/cache	DELETE	Invalidate by tenant, doc set, or prefix

Data Model

Entity	Key fields	Notes
exact_entries	hash(prompt\|model\|params), response, ttl	Redis
semantic_entries	id, embedding, response_ref, acl_key	ANN index per tenant
invalidations	source_version, ts	Bump when KB changes

High-level Design

Lookup path

Request → normalize prompt → exact Redis hash
              miss → embed → ANN (tenant + acl_key filter)
              sim ≥ τ? → return cached (record soft-hit)
              else → LLM → put exact + semantic
KB update → bump corpus_version → invalidate related keys

Deep Dive: Similarity Thresholds

Tune τ on a labeled paraphrase set; optimize precision over recall
Require matching model_id and tool-enabled flag — never mix modes
For personalized answers, include acl_key / user segment in the partition
Shadow mode: serve LLM always, log would-hit for offline precision

Bottlenecks at Scale

Embed QPS — batch micro-batches of 32; use small embedding model
Stale answers after KB edit — wire invalidation to ingest pipeline
ANN recall latency — HNSW in-memory per tenant shard

Failure Modes & Monitoring

SLO	Target	Alert
Hit rate	≥ 40%	Warn if < 30%
False hit (sampled)	< 1%	Page if > 2%
Lookup p99	< 30ms	Warn if > 50ms
Stale after invalidate	< 60s	Warn

AI & FDE Interview Room