Case study

Design a Notification System

Deliver push, email, and SMS notifications at scale with user preferences and provider failover.

fan-outtemplatesmulti-channel

Requirements

Functional: send notifications via push (FCM/APNs), email, SMS; user channel preferences
Support transactional (password reset) and marketing (promo) with different priorities
Templates with variable substitution; schedule future delivery
Non-functional: transactional p99 delivery < 30 sec; at-least-once with idempotency
Scale: 100M users; 1B notifications/day

Back-of-envelope Estimation

Metric	Calculation	Result
Total/day	1B notifications	Mixed channels
Avg QPS	1B / 86,400	~12K/sec
Peak QPS	5× avg (campaign blast)	~60K/sec
Push (60%)	600M/day	FCM handles burst; batch where possible
Email (30%)	300M/day	~3.5K/sec avg via SendGrid/SES
Storage (logs, 90 days)	1B × 200 B × 90	~18 TB

API Design

Endpoint	Description
POST /v1/notifications	{ userId, templateId, channels[], data, idempotencyKey }
GET /v1/notifications/{id}/status	queued \| sent \| delivered \| failed
PUT /v1/users/{id}/preferences	Channel opt-in/out, quiet hours
POST /v1/templates	Admin: create template per channel

Data Model

Entity	Fields
notification_events	event_id, user_id, template_id, payload, idempotency_key, status, created_at
delivery_attempts	attempt_id, event_id, channel, provider, status, error, ts
user_preferences	user_id, push_enabled, email, sms, quiet_hours_start/end
device_tokens	user_id, platform, token, last_seen

High-level Design

Notification pipeline

Producer ──► API ──► validate prefs ──► Router
                                      │
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
              Push queue        Email queue       SMS queue
                    │                 │                 │
              FCM/APNs          SendGrid/SES        Twilio
                    └─────────────────┴─────────────────┘
                                      │
                              Delivery status DB

Deep Dive: Priority & Rate Limiting

Separate Kafka topics: transactional (priority 0), standard, marketing (priority 2)
Per-user rate limit: max 10 marketing/day; transactional bypasses limit
Quiet hours: queue until window opens unless urgent flag
Idempotency: dedupe table keyed by idempotency_key for 24h

Deep Dive: Provider Retries & DLQ

Exponential backoff: 1s, 5s, 30s, 5min (max 5 attempts). Invalid device token → mark dead, stop retry. Provider 5xx → retry; 4xx permanent → DLQ for manual review.

Batch push

FCM supports 500 tokens per multicast. Group users by template and send in batches to reduce API calls during campaigns.

Failure Modes & Monitoring

SLO	Target
Transactional delivery p99	< 30 sec
End-to-end success rate	> 99% (excluding invalid tokens)
DLQ depth	< 10K events

Alert: provider error rate > 5%, queue lag > 5 min, dedupe collision spike
Dashboard: delivery funnel by channel, template performance, opt-out rate

System Design