learning.lab
Scale & Ops· 29/33

Replication & Keeper

High availability

Plain MergeTree has no idea other servers exist. ReplicatedMergeTree is the same storage engine with one addition: every meaningful operation (a new part appearing, a merge completing) is recorded in a shared log that other replicas watch and replay — coordinated through ClickHouse Keeper.

Keeper: the thing that makes replicas agree

Keeper is ClickHouse's built-in replacement for ZooKeeper — a small, separate consensus service (Raft-based) that stores the replication log and coordinates leader election for tasks like deciding which replica performs a given merge. ClickHouse the database doesn't do consensus itself; it delegates that entirely to Keeper.

an INSERT into a 3-replica table
123
Keeper quorum
replication log · consensus
Replica 1ReplicatedMergeTreereceives INSERT
Replica 2ReplicatedMergeTreefetches new part
Replica 3ReplicatedMergeTreefetches new part

The INSERT only needs to land on one replica. Keeper records the operation in a replication log; every other replica pulls the log, fetches the missing part, and stays consistent — independent of how many replicas exist.

Declaring a replicated table

CREATE TABLE events
(
    user_id UInt64,
    event   String
)
ENGINE = ReplicatedMergeTree(
    '/clickhouse/tables/{shard}/events', -- path in Keeper
    '{replica}'                          -- this replica's name
)
ORDER BY user_id;

The Keeper path identifies which replicas belong to the same logical table — every replica registers itself under the same path, using its own unique replica name. This is the piece that's easy to get wrong: a typo in the path silently creates an unrelated, unreplicated table instead of joining the group.

What replication actually guarantees

  • A write acknowledged by one replica will eventually exist on all of them — via the shared log, not by re-sending the insert to every replica.
  • Any replica can serve reads independently; ClickHouse doesn't require quorum reads by default, which is faster but means a read can briefly lag the very latest write on a different replica.
  • If a replica goes down and comes back, it catches up by replaying the log — it doesn't need a full manual resync unless it was down long enough to fall outside Keeper's log retention.
Production note
Keeper itself needs to run as a quorum (typically 3 nodes) to tolerate a single node failure — a single Keeper instance is a single point of failure for the entire cluster's coordination, even if the data replicas themselves are numerous.
Why it exists
Replication (availability, via ReplicatedMergeTree) and sharding (capacity, via Distributed) are independent axes. A production cluster's typical shape is: each shard is a small group of replicas, and a Distributed table routes across shards while each shard tolerates individual node failure via replication.