13.6 Logging

"Like many applications, the Kafka broker will FILL DISKS with log messages IN MINUTES if you let it. In order to get useful information from logging, it is important to enable the right loggers at the right levels."

Baseline: "By simply logging all messages at the INFO level, you will capture a significant amount of important information."

Two loggers to separate into their own files, both at INFO:

kafka.controller (INFO)

“provides messages Specifically regarding the cluster controller. At any time, Only one broker will be the controller, and therefore Only one broker will be writing to this logger.”

Contents: “topic Creation and Modification, broker Status changes, and cluster activities such as Preferred replica elections and Partition moves.”

kafka.server.ClientQuotaManager (INFO)

“messages related to Produce and consume quota activities. While this is useful information, It is better to not have it in the main broker log file.”

💡 The log-compaction loggers — covering a genuine monitoring blind spot

*"It is also helpful to log information regarding the status of the log compaction threads. THERE IS NO SINGLE METRIC TO SHOW THE HEALTH OF THESE THREADS, AND IT IS POSSIBLE FOR FAILURE IN COMPACTION OF A SINGLE PARTITION TO HALT THE LOG COMPACTION THREADS ENTIRELY, AND SILENTLY.

Enabling kafka.log.LogCleaner, kafka.log.Cleaner, and kafka.log.LogCleanerManager at the DEBUG level will output information about the status of these threads, including information about each partition being compacted, including the size and number of messages in each.

Under normal operations, THIS IS NOT A LOT OF LOGGING, WHICH MEANS THAT IT CAN BE ENABLED BY DEFAULT WITHOUT OVERWHELMING YOU."*

Enable the three DEBUG loggers — kafka.log.LogCleaner, kafka.log.Cleaner, kafka.log.LogCleanerManager — by default. It's cheap.

For debugging only:

kafka.request.logger at DEBUG or TRACE

DEBUG

“connection End points, request Timings, and Summary information”

TRACE

“also Topic and partition information — nearly all request information Short of the message payload itself”

⚠ “At Either level, this logger generates a Significant amount of data, and It is not recommended to enable it unless necessary for debugging.”

(Ch. 11 §7 showed what one of these lines contains — it's simultaneously an audit record and a full latency trace.)

---