Production Failure Scenarios
Too many parts, OOM, replication lag
Every concept in this module has a corresponding way to misuse it. This page collects the failures that actually happen in production ClickHouse deployments, in symptom → cause → fix form, so they're recognizable the first time you hit them instead of the third.
"Too many parts"
Symptom: inserts start failing with an error mentioning too many parts, often during a traffic spike.
Cause: every INSERT creates at least one new part (see
Parts & Background Merges). If inserts
arrive in small, frequent batches — or one row at a time — parts pile
up faster than background merges can combine them, and ClickHouse
starts rejecting new inserts as a safety valve rather than letting the
part count grow unbounded.
Fix: batch inserts client-side into the thousands of rows per
INSERT, or route through the async insert mechanism described in
Async Inserts & Batching. This is
almost always an ingestion-pattern problem, not a server capacity
problem — adding hardware doesn't fix it.
Query killed for memory (OOM)
Symptom: a query fails with a memory limit exceeded error, often a
large JOIN or GROUP BY with high cardinality.
Cause: the query's intermediate state (hash tables for the join or
aggregation) grew past max_memory_usage, covered in
Server Tuning. This is ClickHouse
protecting the rest of the server, not a bug.
Fix: reduce what the query has to hold in memory — filter earlier, aggregate in stages, or check whether a join is unintentionally broadcasting a large table. Raising the memory limit is sometimes correct, but treat it as a last resort, not the first thing to try.
An accidental full-table rewrite
Symptom: a routine mutation
(ALTER TABLE ... UPDATE/DELETE) or an OPTIMIZE TABLE ... FINAL run
on a large table turns into a long-running operation that saturates
disk I/O and is awkward to cancel cleanly.
Cause: both operations work by rewriting entire parts, not patching rows in place — exactly as described in Mutations and Parts & Background Merges. On a multi-terabyte table, that's a multi-terabyte rewrite, however small the actual change.
Fix: scope mutations with the narrowest WHERE possible (ideally
aligned to a partition), and treat OPTIMIZE ... FINAL as a
deliberate, scheduled maintenance operation on large tables — never
a routine one. If rows genuinely need frequent updates, reconsider
the schema (e.g. a ReplacingMergeTree) instead of relying on
mutations.
A replica falling behind
Symptom: reads from one replica return noticeably older data than another, or the cluster feels inconsistent depending on which node a query happens to hit.
Cause: as described in Replication & Keeper, a replica catches up by replaying a log coordinated through Keeper. If that replica is undersized, network-constrained, or busy with its own merges, its replication queue grows and it drifts further behind.
Fix: watch replication queue size as a first-class metric (see Monitoring), not an afterthought — a queue that only grows is an early warning, not a one-time blip. A persistently lagging replica usually needs more resources or fewer competing responsibilities, not a restart.
Disk full from data that was never supposed to stick around
Symptom: the server refuses inserts, or merges start failing, because a data disk is out of space.
Cause: raw, high-volume tables with no TTL policy and no partition-based cleanup keep every row forever by default — ClickHouse won't delete anything on your behalf unless told to.
Fix: add a TTL policy (or a scheduled DROP PARTITION) to every
table that's meant to have a lifecycle, before it fills a disk — not
after. Disk-full is one of the very few ClickHouse failure modes
that isn't self-healing once it happens; merges themselves need free
space to write their output.
First three things to check when something's wrong
system.query_logfor the slow or failing query itself — duration, rows read, memory used (see System Tables).system.partsandsystem.mergesfor part counts and merge pressure on the table involved.- Your Prometheus/Grafana dashboards (see Monitoring) for the trend leading up to the incident — a replication queue or disk usage graph almost always shows the problem building minutes or hours before it became visible.
Almost every failure on this page is a consequence of one of two things: writing to ClickHouse the way you'd write to an OLTP database (tiny frequent writes, row-by-row updates), or letting data accumulate with no lifecycle plan. Both are schema and ingestion-pattern problems, not scaling problems — which is why they show up on a single local instance just as easily as on a large cluster.