12.6 Dumping log segments — `kafka-dump-log.sh`
💡 This output is Chapter 6 §6.5's batch header, made concrete.
Why: "perhaps because you ended up with a 'POISON PILL' message in your topic that is CORRUPTED and your consumer cannot handle it. ... This will allow you to VIEW INDIVIDUAL MESSAGES WITHOUT NEEDING TO CONSUME AND DECODE THEM."
Path convention: <log.dirs>/<topic-name>-<partition>/
Metadata only:
kafka-dump-log.sh --files /tmp/kafka-logs/my-topic-0/00000000000000000000.log
Starting offset: 0
baseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1 lastSequence: -1
producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0
isTransactional: false isControl: false position: 0
CreateTime: 1623034799990 size: 77 magic: 2
compresscodec: NONE crc: 1773642166 isvalid: true
...With payloads — add --print-data-log:
kafka-dump-log.sh --files .../00000000000000000000.log --print-data-log
baseOffset: 0 lastOffset: 0 count: 1 ...
| offset: 0 CreateTime: 1623034799990 keysize: -1 valuesize: 9
sequence: -1 headerKeys: [] payload: Message 1
...💡 This output is Chapter 6 §6.5's batch header, made concrete. Map them:
| Field | What it is |
|---|---|
magic: 2 | the v2 message format |
baseOffset / lastOffset | “offset of the first message + difference to last” |
partitionLeaderEpoch | “epoch of the leader that received the batch” — used for truncation after leader election |
producerId / producerEpoch / baseSequence | the IDEMPOTENT PRODUCER fields (Ch. 8 §2.3) |
isTransactional | transaction attribute bit (Ch. 8) |
isControl | a CONTROL BATCH (commit/abort marker, Ch. 8 §3.5) |
compresscodec | compression type attribute |
crc / isvalid | the checksum |
keysize: -1 | NULL KEY |
headerKeys: [] | record headers (Ch. 3 §10) |
Index validation options:
"The index is used for finding messages within a log segment, and if corrupted, will cause ERRORS IN CONSUMPTION. Validation is performed WHENEVER A BROKER STARTS UP IN AN UNCLEAN STATE (i.e., it was not stopped normally), but it can be performed manually as well."
| Option | Depth |
|---|---|
--index-sanity-check | "just check that the index is in a usable state" |
--verify-index-only | "check for mismatches in the index without printing out all the index entries" |
--value-decoder-class | "allows serialized messages to be deserialized by passing in a decoder" |
(Ch. 6 §6.6: indexes are derived data — safe to delete, auto-regenerated. This tool tells you whether you need to.)