Chapter
System Design Fundamentals
Scalability, load balancing, caching, and CDNs — the building blocks every design starts with.
Scalability & Performance
Scalable systems handle growth in users, data, and traffic without proportional cost or latency spikes. Two metrics dominate interviews: latency (time per request) and throughput (requests per second). Under load they often trade off — batching raises throughput but adds queuing delay.
| Strategy | When to use | Limit |
|---|---|---|
| Vertical scaling (bigger machine) | Early stage, simple stack | Hardware ceiling, single point of failure |
| Horizontal scaling (more nodes) | Stateless app tier, read-heavy DB | Requires load balancing & data partitioning |
| Caching | Repeated reads of hot data | Invalidation complexity |
| Async processing | Slow or spiky downstream work | Eventual consistency |
- Stateless services scale linearly — push session/state to Redis or DB
- Shared-nothing architecture avoids lock contention across nodes
- Amdahl's Law: speedup is limited by the serial fraction of work
- Little's Law: L = λW — queue depth = arrival rate × service time
Scale OUT (horizontal) Scale UP (vertical)
App-1 App-2 App-3 [ Bigger CPU/RAM ]
\ | / |
Load Balancer Single node
| |
Shared DB Shared DB
(partition when DB becomes bottleneck)Interview signal
Always state whether the bottleneck is CPU, memory, disk I/O, or network — then pick the fix that attacks that resource. Mention whether the workload is read-heavy, write-heavy, or compute-bound.
Performance Metrics & Bottlenecks
Performance is measured at multiple layers. User-facing SLIs include p50/p95/p99 latency, error rate, and availability. Internal metrics (CPU, memory, disk IOPS, network bandwidth) explain why SLIs degrade. Always tie metrics to the critical user path.
| Metric | What it measures | Interview use |
|---|---|---|
| Latency (p50/p99) | Response time distribution | Tail latency drives UX; p99 often 10× p50 |
| Throughput (QPS/RPS) | Requests handled per second | Capacity planning anchor |
| Error rate | Failed requests / total | SLO budgets, circuit breaker triggers |
| Saturation | Resource utilization % | CPU > 70% sustained → scale soon |
| Apdex / SLI | User-perceived quality | Maps latency to satisfaction score |
- Profile the critical path — trace one request end-to-end
- Identify the slowest hop (often DB, external API, or lock contention)
- Check for N+1 queries, missing indexes, or synchronous fan-out
- Validate with load test; watch tail latency under peak, not just average
USE method
For each resource: Utilization (busy %), Saturation (queue depth), Errors. Applies to CPU, memory, disk, and network. Faster than guessing in interviews.
Common bottleneck traps
Hot keys in cache/DB, connection pool exhaustion, GC pauses on large heaps, DNS TTL misconfiguration, and thundering herd on cache expiry. Name at least two in a deep dive.
Load Balancing L4/L7
A load balancer distributes traffic across backend servers. Layer 4 (transport) balancers route by IP address and TCP/UDP port — fast, protocol-agnostic, but blind to HTTP content. Layer 7 (application) balancers inspect HTTP headers, paths, and cookies for intelligent routing, TLS termination, and WAF integration.
| Layer | Routes on | Pros | Cons |
|---|---|---|---|
| L4 (TCP/UDP) | IP + port | Low latency, any protocol | No path-based routing |
| L7 (HTTP) | URL, headers, cookies | Content routing, TLS offload | Higher CPU per connection |
| DNS load balancing | Geo / weighted records | Global distribution | TTL delay on failover |
| Client-side LB | Service discovery list | No single LB hop | Client complexity |
Client → DNS → CDN (static assets)
↓
Load Balancer (L7)
/ | \
App-1 App-2 App-3 (stateless)
↓
Cache / DB / Queue- Reverse proxy (nginx, HAProxy, ALB) sits in front of your servers
- Forward proxy sits in front of clients (corporate egress) — different use case
- TLS termination at LB reduces CPU on app servers
- Connection draining gracefully removes nodes during deploys
When to mention each layer
Start with L7 for HTTP APIs (path routing, auth headers). Mention L4 for WebSockets, gRPC over TCP, or gaming backends. DNS/Geo LB for multi-region active-active.
LB Algorithms & Health Checks
Load balancing algorithms decide which backend receives each request. Health checks continuously probe backends and remove unhealthy nodes from rotation. Together they determine availability, fairness, and stickiness under failure.
| Algorithm | Behavior | Best for |
|---|---|---|
| Round-robin | Cycle through backends equally | Homogeneous, short requests |
| Weighted round-robin | Proportional to server capacity | Mixed instance sizes |
| Least connections | Route to fewest active connections | Long-lived or variable-duration requests |
| Consistent hashing | Hash key → ring position | Cache affinity, minimal reshuffle on node change |
| IP hash | Client IP → fixed backend | Simple session stickiness (fragile on NAT) |
- Active health checks: LB sends HTTP/TCP probes on an interval
- Passive health checks: mark unhealthy after N consecutive errors
- Grace period on startup — don't route until app is ready (readiness probe)
- Separate liveness (restart) vs readiness (remove from LB) in Kubernetes
Sticky sessions
Session affinity via cookie or consistent hashing keeps user state on one node. Prefer external session store (Redis) over stickiness — stickiness breaks on node loss and complicates deploys.
Health check pitfalls
Checking /health that only returns 200 without verifying DB connectivity causes black-hole routing. Deep health checks should validate critical dependencies with timeouts.
Caching Patterns
Caches store copies of expensive-to-compute or expensive-to-fetch data closer to the reader. Placement matters: browser cache, CDN edge, application in-memory, distributed cache (Redis), and database buffer pool each serve different TTL and consistency needs.
| Pattern | Behavior | Tradeoff |
|---|---|---|
| Cache-aside | App reads cache; on miss, read DB and populate | Simple; stale risk on writes |
| Read-through | Cache library fetches from DB on miss | Centralized logic; vendor lock-in |
| Write-through | Write updates cache and DB synchronously | Consistent cache; slower writes |
| Write-back (write-behind) | Write to cache; flush to DB async | Fast writes; data loss risk on crash |
| Refresh-ahead | Proactively refresh before expiry | Smooth latency; wasted work on cold keys |
- Cache hit ratio = hits / (hits + misses) — target > 90% for hot paths
- Key design: namespace:entity:id (e.g. user:profile:12345)
- Invalidate on write vs TTL-only — pick based on staleness tolerance
- Local (in-process) cache + distributed cache = two-tier; watch consistency
What to cache
Cache read-heavy, relatively stable data: user profiles, product catalogs, config flags, computed feeds. Avoid caching highly personalized or rapidly changing data unless TTL is very short.
CDNs & Edge Caching
Content Delivery Networks are geographically distributed edge servers that cache static and cacheable dynamic content close to users. They reduce origin load, cut latency, and absorb DDoS traffic. Major providers: CloudFront, Cloudflare, Fastly, Akamai.
User (Tokyo)
↓
Edge PoP (cache HIT → return)
↓ (MISS)
Shield / Mid-tier PoP
↓ (MISS)
Origin (S3 / App server)
↓
Populate edge cache (Cache-Control headers)| Content type | Cache strategy | Header example |
|---|---|---|
| Static assets (JS/CSS/img) | Long TTL + fingerprinted URLs | Cache-Control: max-age=31536000, immutable |
| HTML pages | Short TTL or stale-while-revalidate | max-age=60, stale-while-revalidate=300 |
| API responses | Selective; often no CDN | Cache-Control: private, no-store |
| Video (HLS/DASH) | Segment caching at edge | Per-segment TTL ~ segment duration |
- Cache key = URL + Vary headers (Accept-Encoding, Accept-Language)
- Purging / cache invalidation via API when content updates urgently
- Dynamic site acceleration (DSR) routes API through optimized edge paths
- Origin shield reduces thundering herd on origin during global miss spikes
Interview shortcut
For media-heavy products (Netflix, YouTube), lead with CDN + object storage. Mention adaptive bitrate streaming and edge caching of video segments separately from metadata API.
Cache Eviction & Stampede
Caches have finite memory. Eviction policies decide which keys to remove when full. A cache stampede (thundering herd) occurs when a hot key expires and thousands of concurrent requests miss cache simultaneously, overwhelming the origin database.
| Eviction policy | Behavior | Use case |
|---|---|---|
| LRU (Least Recently Used) | Evict coldest access time | General purpose; Redis default |
| LFU (Least Frequently Used) | Evict lowest hit count | Stable hot set over time |
| TTL (Time To Live) | Expire after fixed duration | Simple staleness bound |
| Random | Evict arbitrary key | Memcached default; simple |
| ARC / W-TinyLFU | Adaptive LRU+LFU hybrid | High hit ratio in production CDN caches |
- Single-flight / request coalescing: one thread rebuilds; others wait on same key
- Probabilistic early expiration: refresh hot keys before hard TTL
- Stale-while-revalidate: serve stale while async refresh runs
- TTL jitter: spread expirations (base TTL ± random offset)
- Circuit breaker on origin when miss rate spikes abnormally
Cache stampede scenario
Celebrity tweet crashes your site: a fan-out key (trending topic) expires. Mitigate with per-key locks in Redis, pre-warming, and never letting ultra-hot keys expire without a background refresher.
Design a cache layer
Walk through: key schema → TTL policy → eviction → invalidation on write → stampede mitigation. Mention Redis cluster for horizontal scale and hash tags for co-located keys.