learning.lab
Monitoring & Operations· 31/33

Monitoring

Prometheus & Grafana integration

Querying system tables by hand is great for investigating a specific problem right now. It doesn't give you history, alerting, or a dashboard someone can glance at during an incident — that's what a metrics pipeline is for.

The built-in Prometheus exporter

ClickHouse ships a Prometheus-compatible metrics endpoint — enabled in server config and typically served on its own port (or under /metrics on the HTTP port, depending on how it's configured). It exposes the same kinds of numbers already visible in system.metrics, system.asynchronous_metrics, and system.events, in a format Prometheus can scrape on a schedule and retain over time.

the standard path
ClickHouse/metrics endpoint
Prometheusscrapes on an interval
Grafanadashboards + alerts
config.xml (excerpt)
<prometheus>
    <endpoint>/metrics</endpoint>
    <port>9363</port>
    <metrics>true</metrics>
    <events>true</events>
    <asynchronous_metrics>true</asynchronous_metrics>
</prometheus>

What's actually worth watching

A handful of metrics map directly onto concepts already covered in this module, which is the fastest way to build intuition for what "normal" looks like on your own cluster:

  • Background merge counts and merge queue size — the health of the process described in Parts & Background Merges. A queue that only grows means merges can't keep up with insert volume.
  • Replication queue size per replica — directly tied to Replication & Keeper. A replica whose queue keeps growing is falling behind the others.
  • Query count, memory usage, and rejected/failed queries — the live counterparts to what system.query_log and system.processes show after the fact.
  • Disk space per volume — the earliest warning for the disk-full failure mode covered in Production Failure Scenarios.

A second, separate integration path: the Grafana data source

Prometheus/Grafana covers server health metrics. Separately, ClickHouse publishes an official Grafana data source plugin that lets Grafana query ClickHouse directly with SQL — for building dashboards over your actual application data (events, business metrics) rather than server internals. It's easy to conflate the two: one path monitors the database, the other uses the database as a source for your own analytics dashboards.

Production note
Alert on trends, not just thresholds. A replication queue of 500 is alarming on a cluster that's usually near zero, and unremarkable on one that always hovers around a few hundred during normal ingestion bursts. Static thresholds copied from someone else's cluster tend to either miss real problems or page you constantly for nothing.