12.7 Replica verification — `kafka-replica-verification.sh`

Why it's needed — the gap it covers:

"Partition replication works similar to a regular Kafka consumer client: the follower broker starts replicating at the oldest offset and checkpoints the current offset to disk periodically. When replication stops and restarts, it picks up from the last checkpoint. IT IS POSSIBLE FOR PREVIOUSLY REPLICATED LOG SEGMENTS TO GET DELETED FROM A BROKER, AND THE FOLLOWER WILL NOT FILL IN THE GAPS IN THIS CASE."

What the tool does: "fetch messages from ALL the replicas for a given set of topic partitions, check that all messages exist on all replicas, and print out the max lag for given partitions. This process will operate continuously in a loop until canceled."

kafka-replica-verification.sh \
  --broker-list kafka.host1.domain.com:9092,kafka.host2.domain.com:9092 \
  --topic-white-list 'my.*'

2021-06-07 03:28:21,829: verification process is started.
2021-06-07 03:28:51,949: max lag is 0 for partition my-topic-0 at offset 4 among 1 partitions

"you must provide an EXPLICIT comma-separated list of brokers... By default, all topics are validated; however, you may also provide a regular expression."

⚠️ CAUTION: CLUSTER IMPACT AHEAD

"The replica verification tool will have an impact on your cluster SIMILAR TO REASSIGNING PARTITIONS, as it must READ ALL MESSAGES FROM THE OLDEST OFFSET in order to verify the replica. In addition, IT READS FROM ALL REPLICAS FOR A PARTITION IN PARALLEL, so it should be used with caution."

Cost model: (bytes in topic) × (replication factor), read from the oldest offset, in parallel, in a loop.

► This will evict your entire page cache. Use it deliberately, off-peak.