12.2 Consumer groups — `kafka-consumer-groups.sh`

Can "list consumer groups, describe specific groups, delete consumer groups or specific group info, or reset consumer group offset information."

ZOOKEEPER-BASED CONSUMER GROUPS

"In older versions, consumer groups could be managed in ZooKeeper. This behavior was DEPRECATED in versions 0.11.0.* and later, and old consumer groups are no longer used. Some versions of the provided scripts may still show deprecated --zookeeper connection string commands, but it is NOT RECOMMENDED to use them unless you have an old environment."

2.1 List and describe

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
# console-consumer-95554     ← ad hoc consumers appear as
# console-consumer-9581      ←   console-consumer-<generated_id>
# my-consumer

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-consumer

GROUP        TOPIC     PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID          HOST       CLIENT-ID
my-consumer  my-topic  0         2              4              2   consumer-1-029af...  /127.0.0.1 consumer-1
my-consumer  my-topic  1         2              3              1   consumer-1-029af...  /127.0.0.1 consumer-1
my-consumer  my-topic  2         2              3              1   consumer-2-42c1a...  /127.0.0.1 consumer-2

The fields, precisely defined (Table 12-1):

(Note LOG-END-OFFSET is the high-water mark, not the log-end offset in the Ch. 6 §5.6 sense — so this LAG is measured against what's readable, which is the right thing.)

2.2 Delete a group

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-consumer
# Deletion of requested consumer groups ('my-consumer') was successful.

"This will remove the entire group, including ALL STORED OFFSETS for ALL topics the group is consuming. To perform this action, ALL CONSUMERS IN THE GROUP SHOULD BE SHUT DOWN as the group must not have any active members. If you attempt to delete a group that is not empty, an error stating 'The group is not empty' will be thrown and NOTHING WILL HAPPEN."

💡 "It is also possible to use the same command to delete offsets for a SINGLE TOPIC that the group is consuming without deleting the entire group by adding the --topic argument."

2.3 Offset management

Why: "useful for resetting the offsets for a consumer when there is a problem that requires messages to be reread, or for advancing offsets and skipping past a message that the consumer is having a problem with (e.g., if there is a badly formatted message that the consumer cannot handle)."

Export (via --dry-run)

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --export --group my-consumer --topic my-topic \
  --reset-offsets --to-current --dry-run > offsets.csv

cat offsets.csv
# my-topic,0,8905
# my-topic,1,8915
# ...

CSV format: <topic-name>,<partition-number>,<offset>

⚠️ "Running the same command WITHOUT the --dry-run option will RESET THE OFFSETS COMPLETELY, so be careful."

That's a genuinely dangerous ergonomic: the export command and the destructive reset command differ by one flag.

Import

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --reset-offsets --group my-consumer \
  --from-file offsets.csv --execute

  TOPIC       PARTITION  NEW-OFFSET
  my-topic    0          8905
  ...

The recommended workflow: "A common practice is to export the current offsets, MAKE A COPY OF THE FILE (so that you preserve a backup), and EDIT THE COPY to replace the offsets with the desired values."

⚠️ STOP CONSUMERS FIRST

"Before performing this step, it is important that ALL consumers in the group are STOPPED. THEY WILL NOT READ THE NEW OFFSETS IF THEY ARE WRITTEN WHILE THE CONSUMER GROUP IS ACTIVE. THE CONSUMERS WILL JUST OVERWRITE THE IMPORTED OFFSETS."

(Same constraint as Ch. 5 §6.4 — and there it explains why: groups only read offsets on assignment or startup.)