4.2 Rebalancing — the mechanism, and why it hurts
Rebalance protocol
- 2
- rebalance rounds
Only 3 of 12 partitions pause (25%). The rest keep consuming throughout. The cost is two rebalance rounds instead of one, which is almost always the better trade.
When does a rebalance happen?
- A new consumer joins the group → it starts consuming partitions previously owned by another consumer
- A consumer shuts down or crashes → it leaves; its partitions go to a remaining consumer
- The topics the group consumes are modified — e.g. an administrator adds new partitions
"Moving partition ownership from one consumer to another is called a rebalance. Rebalances are important because they provide the consumer group with high availability and scalability ... but in the normal course of events they can be fairly undesirable."
2.1 Eager rebalance — "stop the world"
“During an eager rebalance, all consumers stop consuming, give up their ownership of all partitions, rejoin the consumer group, and get a brand-new partition assignment. This is essentially a short window of unavailability of the entire consumer group.”
"During an eager rebalance, all consumers stop consuming, give up their ownership of all partitions, rejoin the consumer group, and get a brand-new partition assignment. This is essentially a short window of unavailability of the entire consumer group. The length of the window depends on the size of the consumer group as well as on several configuration parameters."
2.2 Cooperative (incremental) rebalance
- Only the REASSIGNED partitions paused. No “stop the world.”
- “May take a few iterations until a stable assignment is achieved.”
"Cooperative rebalances (also called incremental rebalances) typically involve reassigning only a small subset of the partitions from one consumer to another, and allowing consumers to continue processing records from all the partitions that are not reassigned. ... This is especially important in large consumer groups where rebalances can take a significant amount of time."
Sequence: (1) group leader informs all consumers they will lose ownership of a subset; consumers stop consuming those and give up ownership. (2) Group leader assigns the now-orphaned partitions to new owners.
2.3 Heartbeats, the group coordinator, and the two death-detection paths
“As long as the consumer is sending heartbeats at regular intervals, it is assumed to be alive. If the consumer stops sending heartbeats for long enough, its session will timeout and the group coordinator will consider it dead and trigger a rebalance.”
"As long as the consumer is sending heartbeats at regular intervals, it is assumed to be alive. If the consumer stops sending heartbeats for long enough, its session will timeout and the group coordinator will consider it dead and trigger a rebalance."
The two failure paths — and why there must be two:
| Path | Detects | Config | Why it exists |
|---|---|---|---|
| Session timeout (missed heartbeats) | The process is dead / unreachable | session.timeout.ms + heartbeat.interval.ms | The obvious case |
| Poll interval timeout | The main thread is stuck while the background heartbeat thread is still happily heartbeating | max.poll.interval.ms | "There is a possibility that the main thread consuming from Kafka is deadlocked, but the background thread is still sending heartbeats. This means that records from partitions owned by this consumer are not being processed." |
Crash vs clean shutdown:
The coordinator needs “a few seconds without heartbeats to decide it is dead,” and during those seconds no messages are processed from the dead consumer's partitions.
This is why close() matters operationally, not just for tidiness. See §7.
2.4 How partition assignment actually works
- Consumer sends JoinGroup request to the group coordinator
- The first consumer to join becomes the group leader
- Leader receives, from the coordinator, the list of All consumers in the group (all that sent a heartbeat recently = considered alive)
- Leader runs an implementation of
PartitionAssignorto decide which partitions go to which consumer - Leader sends the assignment list to the
GroupCoordinator - Coordinator sends the info to all consumers
- Each consumer only sees its own assignment — the leader is the only client process with the full picture
- This whole process repeats on Every rebalance
Design note worth appreciating: assignment logic runs on a client, not the broker. That's why you can plug in your own PartitionAssignor without touching the cluster — and why the leader's Kafka client version determines which strategies are available.