Case study

Design a Metrics & Monitoring System

Ingest billions of time-series data points daily and power dashboards, alerts, and SLO tracking.

time-seriesaggregationalerting

Requirements

Functional: services push metrics (counter, gauge, histogram); query by time range and labels
Dashboards, alerting rules (threshold, anomaly), SLO burn-rate alerts
Non-functional: ingestion 1M data points/sec; query p99 < 2 sec for 24h range
Retention: 15 sec granularity for 7 days; 5 min rollups for 1 year
Scale: 100K services; 10K metrics each avg

Back-of-envelope Estimation

Metric	Calculation	Result
Ingestion rate	1M points/sec	Given
Raw/day	1M × 86,400 × 16 B	~1.3 TB/day uncompressed
After compression	~10×	~130 GB/day
Year storage (rollups)	130 GB × 7 + downsampled	~2 PB tiered
Query QPS	10K concurrent dashboards / 30 sec refresh	~300/sec

API Design

Endpoint	Description
POST /v1/write	Prometheus remote write or { metrics[] } batch
GET /v1/query?promql=	Instant query
GET /v1/query_range?promql=&start=&end=&step=	Range query
POST /v1/alerts/rules	Define alert rule (PromQL + threshold + duration)

Data Model

Layer	Schema
Time series ID	hash(metric_name + sorted labels)
Raw samples	(series_id, timestamp, value) — 15 sec resolution
Rollups	(series_id, window_start, min, max, avg, count) — 5 min/hour/day
Alert state	rule_id, status, last_eval, firing_since

High-level Design

Metrics pipeline

Agents ──► Kafka (ingestion) ──► Stream Aggregators
                                        │
                                  Time-series DB (Cassandra/M3DB)
                                        │
              Query API ◄──► Cache (hot queries) ──► Grafana dashboards
                                        │
                              Alert Evaluator ──► PagerDuty

Deep Dive: Ingestion & Downsampling

Agents batch samples every 15 sec; push to regional Kafka partition by series_id hash
Stream workers write raw to hot storage (7 days) and compute 5-min rollups async
Rollups compact into hour/day tiers for long-term retention
Cardinality control: drop high-cardinality labels; limit 1M active series per tenant

Deep Dive: Alerting & SLOs

Alert evaluator runs PromQL rules every 30 sec. Multi-window burn rate for SLOs: 14.4× burn over 1h page; 6× over 6h ticket. Group related alerts; inhibit duplicate pages during incident.

Query performance

Pre-aggregate common dashboard queries. Cache query results for 30 sec. Push down aggregations to storage layer.

Failure Modes & Monitoring

SLO	Target
Ingestion lag	< 30 sec p99
Query p99 (24h range)	< 2 sec
Alert delivery	< 1 min from condition true

Dogfood: monitor the monitoring system. Alert on Kafka consumer lag, dropped samples, query error rate, cardinality explosion.

System Design