6.1 Cluster membership — ZooKeeper ephemeral nodes
Note the three causes lumped together: stopped, network partition, long GC pause.
Mechanics:
- Every broker has a unique ID — "either set in the broker configuration file or automatically generated."
- Every time a broker process starts, it registers itself with its ID in ZooKeeper by creating an ephemeral node.
- Start another broker with the same ID → error, because "we already have a ZooKeeper node for the same broker ID."
How a broker "disappears":
"When a broker loses connectivity to ZooKeeper (usually as a result of the broker stopping, but this can also happen as a result of network partition or a long garbage-collection pause), the ephemeral node... will be automatically removed from ZooKeeper. Kafka components watching the list of brokers will be notified that the broker is gone."
Note the three causes lumped together: stopped, network partition, long GC pause. From ZooKeeper's perspective they are indistinguishable — which is exactly why Ch. 2's GC tuning is a cluster stability concern, not just a latency concern.
💡 Broker IDs outlive broker nodes — and this is a feature
"Even though the node representing the broker is gone when the broker is stopped, the broker ID still exists in other data structures. For example, the list of replicas of each topic contains the broker IDs for the replica. This way, if you completely lose a broker and start a brand-new broker with the ID of the old one, it will immediately join the cluster in place of the missing broker with the same partitions and topics assigned to it."
It must then replicate the data as a follower, but the assignment is inherited for free.
This is the recovery pattern for a dead broker: reuse the ID. It also explains Ch. 2's advice to derive broker.id from the hostname — ID↔host mapping is load-bearing during exactly this operation.