Production Failure Scenarios

Every concept in this module has a corresponding way to misuse it. This page collects the failures that actually happen in production ClickHouse deployments, in symptom → cause → fix form, so they're recognizable the first time you hit them instead of the third.

"Too many parts"

Symptom: inserts start failing with an error mentioning too many parts, often during a traffic spike.

Cause: every INSERT creates at least one new part (see Parts & Background Merges). If inserts arrive in small, frequent batches — or one row at a time — parts pile up faster than background merges can combine them, and ClickHouse starts rejecting new inserts as a safety valve rather than letting the part count grow unbounded.

Common mistake

Fix: batch inserts client-side into the thousands of rows per INSERT, or route through the async insert mechanism described in Async Inserts & Batching. This is almost always an ingestion-pattern problem, not a server capacity problem — adding hardware doesn't fix it.

Query killed for memory (OOM)

Symptom: a query fails with a memory limit exceeded error, often a large JOIN or GROUP BY with high cardinality.

Cause: the query's intermediate state (hash tables for the join or aggregation) grew past max_memory_usage, covered in Server Tuning. This is ClickHouse protecting the rest of the server, not a bug.

Common mistake

Fix: reduce what the query has to hold in memory — filter earlier, aggregate in stages, or check whether a join is unintentionally broadcasting a large table. Raising the memory limit is sometimes correct, but treat it as a last resort, not the first thing to try.

An accidental full-table rewrite

Symptom: a routine mutation (ALTER TABLE ... UPDATE/DELETE) or an OPTIMIZE TABLE ... FINAL run on a large table turns into a long-running operation that saturates disk I/O and is awkward to cancel cleanly.

Cause: both operations work by rewriting entire parts, not patching rows in place — exactly as described in Mutations and Parts & Background Merges. On a multi-terabyte table, that's a multi-terabyte rewrite, however small the actual change.

Common mistake

Fix: scope mutations with the narrowest WHERE possible (ideally aligned to a partition), and treat OPTIMIZE ... FINAL as a deliberate, scheduled maintenance operation on large tables — never a routine one. If rows genuinely need frequent updates, reconsider the schema (e.g. a ReplacingMergeTree) instead of relying on mutations.

A replica falling behind

Symptom: reads from one replica return noticeably older data than another, or the cluster feels inconsistent depending on which node a query happens to hit.

Cause: as described in Replication & Keeper, a replica catches up by replaying a log coordinated through Keeper. If that replica is undersized, network-constrained, or busy with its own merges, its replication queue grows and it drifts further behind.

Common mistake

Fix: watch replication queue size as a first-class metric (see Monitoring), not an afterthought — a queue that only grows is an early warning, not a one-time blip. A persistently lagging replica usually needs more resources or fewer competing responsibilities, not a restart.

Disk full from data that was never supposed to stick around

Symptom: the server refuses inserts, or merges start failing, because a data disk is out of space.

Cause: raw, high-volume tables with no TTL policy and no partition-based cleanup keep every row forever by default — ClickHouse won't delete anything on your behalf unless told to.

Common mistake

Fix: add a TTL policy (or a scheduled DROP PARTITION) to every table that's meant to have a lifecycle, before it fills a disk — not after. Disk-full is one of the very few ClickHouse failure modes that isn't self-healing once it happens; merges themselves need free space to write their output.

First three things to check when something's wrong

system.query_log for the slow or failing query itself — duration, rows read, memory used (see System Tables).
system.parts and system.merges for part counts and merge pressure on the table involved.
Your Prometheus/Grafana dashboards (see Monitoring) for the trend leading up to the incident — a replication queue or disk usage graph almost always shows the problem building minutes or hours before it became visible.

Why it exists

Almost every failure on this page is a consequence of one of two things: writing to ClickHouse the way you'd write to an OLTP database (tiny frequent writes, row-by-row updates), or letting data accumulate with no lifecycle plan. Both are schema and ingestion-pattern problems, not scaling problems — which is why they show up on a single local instance just as easily as on a large cluster.