Case study

Design a Chat System (WhatsApp-style)

Real-time 1:1 and group messaging for 50M DAU with delivery receipts and offline sync.

WebSocketpresenceordering

Requirements

Functional: 1:1 and group chat; text, images, read receipts; message history
Online/offline presence; typing indicators
Non-functional: message delivery p99 < 500ms when online; ordering per conversation
At-least-once delivery; clients dedupe by message_id
Scale: 50M DAU; 1B messages/day

Back-of-envelope Estimation

Metric	Calculation	Result
Messages/day	1B	Given
Write QPS	1B / 86,400	~12K/sec avg; ~50K peak
Concurrent connections	50M × 20% online	10M WebSockets
Avg message size	200 bytes text; 50 KB media	Media → object store
Storage/year (text only)	1B × 365 × 200 B	~73 TB/year text metadata

API Design

Interface	Description
WS /v1/chat	Bidirectional: send_message, ack, typing, presence
POST /v1/conversations	Create group; add members
GET /v1/conversations/{id}/messages?before=	Paginated history (offline sync)
POST /v1/media/upload	Pre-signed URL for image upload

Data Model

Entity	Partition key	Fields
messages	conversation_id	msg_id, sender_id, body, seq_num, ts, status
conversations	conversation_id	type (1:1/group), member_ids[], created_at
user_inbox	user_id	conversation_id, last_msg_ts, unread_count
presence	user_id (Redis)	status, last_seen, connection_gateway_id

High-level Design

Real-time messaging

Client ◄──WebSocket──► Gateway (sticky by user_id)
                              │
                    Chat Service ◄──► Message Queue
                              │
                    Cassandra (messages by conversation_id)
                    Redis (presence, seq counters)

Gateway maintains WebSocket and routes to user's active session. Chat service assigns monotonic seq_num per conversation, persists, publishes to recipient's gateway via internal bus. Offline users fetch history on reconnect; push via FCM/APNs.

Deep Dive: Message Ordering

Per-conversation sequence counter in Redis (INCR) — single writer per conversation
Client displays messages sorted by seq_num
Out-of-order network delivery: buffer until gap filled or timeout
Group messages: one seq stream per group; fan-out to each member's inbox

Deep Dive: Group Chat Fan-out

For a 256-member group, one write to message store + 256 inbox updates. Batch inbox updates via queue workers. Large groups (> 256) may use read-receipt aggregation and lazy member notification.

1:1 vs group

1:1 chats can use two-user conversation_id = hash(sorted(userA, userB)). Groups need explicit conversation_id and membership ACL checks.

Failure Modes & Monitoring

SLO	Target
Online delivery p99	< 500ms
Gateway connection success	> 99.5%
Message loss	0 (durability via queue ack)

Gateway failover: client reconnect with exponential backoff; resume from last seq
Monitor: connections per gateway, cross-DC latency, queue lag, seq gaps reported by clients

System Design