Monitoring basics: metrics, Prometheus, Grafana and alerts

Monitoring means collecting numbers about your system over time, showing them on dashboards and alerting you when something is wrong. Without it, you learn about outages from your users.

What to measure

A useful starting set is the four golden signals from Google's SRE book:

• Latency: how long requests take (watch slow percentiles such as p95 and p99, not only the average).
• Traffic: how many requests per second.
• Errors: the rate of failed requests.
• Saturation: how full the resources are (CPU, memory, disk, connections).

For a server also watch disk space, memory, load, and whether key services and certificates are alive and not about to expire.

Metrics, logs and traces

• Metrics: numbers over time. Cheap, good for trends and alerts.
• Logs: detailed events. Good for finding out why.
• Traces: the path of one request across services.

Start with metrics, then add the others when you need them.

How Prometheus and Grafana fit

• Prometheus is a time-series database that pulls (scrapes) metrics from HTTP endpoints, usually /metrics, at a regular interval. It stores them and evaluates alert rules.
• Exporters expose metrics for things that do not do it themselves. node_exporter covers Linux machine metrics.
• Grafana reads from Prometheus and draws dashboards.
• Alertmanager receives alerts from Prometheus, groups them and sends them to email, Slack or another channel.

A small Prometheus config

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["localhost:9100"]
  - job_name: myapp
    static_configs:
      - targets: ["localhost:3000"]

Run node_exporter on port 9100, point Prometheus at it, add Prometheus as a data source in Grafana and import a ready-made node dashboard.

Exposing metrics from your app

Most languages have a Prometheus client library. Expose counters (requests, errors) and histograms (request duration) on /metrics.

Keep that endpoint private: bind it to localhost or an internal network, or protect it, because it can reveal internal details.

Alerts worth having

An example rule that fires when a target has been down for 2 minutes:

groups:
  - name: basics
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} is down"

Good first alerts: service down, disk almost full, high error rate, certificate expiring soon, and backups that did not run.

Avoid alert fatigue

• Alert on symptoms users feel (errors, slowness), not on every small cause.
• Use for: so short spikes do not page you.
• Every alert should be actionable. If nobody needs to act, make it a dashboard panel instead.
• Separate urgent pages from low-priority tickets.
• Review and delete noisy alerts regularly.

Do not forget

• Monitor from outside too (an uptime check), because a dead server cannot report itself.
• Protect Grafana and Prometheus with logins and HTTPS and do not leave them open to the internet.
• Set retention so metrics do not fill the disk.
• Keep a short runbook for each alert: what it means and what to do first.

Takeaway: measure latency, traffic, errors and saturation, let Prometheus scrape and Grafana display, alert only on things a person must act on, and also check your site from the outside.

---

Linux & DevOps series, Part 2 (delivery and operations): 3 of 5. Previous: CI/CD with GitHub Actions: test, build and deploy safely.