Design a metrics monitoring and alerting system

Goal: collect numbers (metrics) from thousands of servers and apps, store them cheaply, show them on dashboards and alert people when something is wrong.

Requirements

• Ingest millions of data points per second.
• Query recent data quickly; keep older data cheaply.
• Flexible aggregation (average, percentile, rate) by dimension.
• Reliable alerting with low false alarms.
• The monitoring system itself must be highly available.

Data model

A metric is a time series: a name, a set of labels (key-value tags) and a series of (timestamp, value) points.

Example: http_requests_total{service="api", status="500", region="dhaka"}.

Collection: push vs pull

• Pull (Prometheus style): the server scrapes each target's /metrics endpoint on a schedule. Easy to see which targets are down; needs service discovery.
• Push (agents, StatsD): apps or agents send metrics to a collector. Better for short-lived jobs and firewalled hosts.

Often a mix is used. A queue (Kafka) can sit in front of ingestion to absorb bursts.

Storage: time-series database

Time-series data is written append-only and read by time range, so special storage works best:

• Group points by series and time block, and use delta and delta-of-delta compression (timestamps and values change little between points). Compression ratios are very high.
• Keep recent data in memory or on fast disk.
• Downsample old data (keep 5-minute averages instead of 10-second points) and apply retention rules to delete the oldest.

Cardinality: the classic trap

Each unique label combination is a separate series. A label with unbounded values, such as user_id or a full URL, can create millions of series and crash the system. Keep labels low-cardinality and put high-cardinality data in logs or traces instead.

Querying and dashboards

A query language computes rates, sums and percentiles across series. Dashboards (like Grafana) run these queries on a refresh interval, so cache common queries and limit time ranges.

Alerting

1. Define rules: for example "error rate above 5% for 5 minutes".
2. An evaluator checks rules periodically.
3. Firing alerts go to an alert manager that groups, deduplicates, silences and routes them (chat, email, phone).

Good alerts are actionable and based on symptoms users feel (errors, latency), with the "for" duration to avoid flapping. Too many noisy alerts cause alert fatigue.

Reliability

Run replicas of collectors and storage, monitor the monitor (a separate simple check), and avoid depending on the system you are watching.

Scaling

Shard series across storage nodes (by hash of the series), run multiple scrapers split by target groups and aggregate with a query layer.

Takeaway: a label-based time-series model, efficient compressed storage with downsampling, strict control of cardinality, and actionable, grouped alerting.

---

System Design series, Part 4 (designing the infrastructure): 4 of 4. Previous: Design a distributed key-value store (Dynamo-style).