11.7 Auditing
"Kafka brokers can be configured to generate comprehensive log4j logs for auditing and debugging."
Two independently configurable loggers:
kafka.authorizer.logger→ authorization loggingkafka.request.logger→ request logging
► They “can be configured independently to customize the log level and retention for audit logging.”
► “Production systems can use frameworks like the Elastic Stack to analyze and visualize these logs.”
⚠️ The log-level asymmetry — this is the important part
| Level | Authorizers generate a line for… |
|---|---|
INFO | every attempted operation for which access was denied |
DEBUG | every operation for which access was granted |
► i.e. at default log levels you see denials but not grants. If you need a full audit trail of successful access, you must enable DEBUG.
Example authorizer log lines:
DEBUG Principal = User:Alice is Allowed Operation = Write from host = 127.0.0.1
on resource = Topic:LITERAL:customerOrders for request = Produce
with resourceRefCount = 1 (kafka.authorizer.logger)
INFO Principal = User:Mallory is Denied Operation = Describe from host = 10.0.0.13
on resource = Topic:LITERAL:customerOrders for request = Metadata
with resourceRefCount = 1 (kafka.authorizer.logger)Request logging:
| Level | What the request log includes |
|---|---|
DEBUG | “includes details of the user principal and client host” |
TRACE | “full details of the request are included” |
DEBUG Completed request:RequestHeader(apiKey=PRODUCE, apiVersion=8,
clientId=producer-1, correlationId=6) -- {acks=-1,timeout=30000,
partitionSizes=[customerOrders-0=15514]},response:{...base_offset=13...},
... totalTime:2.42,requestQueueTime:0.112,localTime:2.15,remoteTime:0.0,
throttleTime:0,responseQueueTime:0.04,sendTime:0.118,
securityProtocol:SASL_SSL,principal:User:Alice,listener:SASL_SSL,
clientInformation:ClientInformation(softwareName=apache-kafka-java,
softwareVersion=2.7.0-SNAPSHOT) (kafka.request.logger)Notice what that single line gives you: the principal, the listener, the security protocol, the client software and version, the correlation ID (Ch. 6 §5), and the full request-pipeline timing breakdown (requestQueueTime, localTime, remoteTime, throttleTime, responseQueueTime, sendTime — the exact stages from Ch. 6's threading diagram). It's simultaneously an audit record and a latency trace.
*"Authorizer and request logs can be analyzed to DETECT SUSPICIOUS ACTIVITIES. METRICS THAT TRACK AUTHENTICATION FAILURES, as well as AUTHORIZATION FAILURE LOGS, can be extremely useful for auditing and provide valuable information IN THE EVENT OF AN ATTACK OR UNAUTHORIZED ACCESS.
For end-to-end auditability and traceability of messages, audit metadata can be included in MESSAGE HEADERS when messages are produced. END-TO-END ENCRYPTION CAN BE USED TO PROTECT THE INTEGRITY OF THIS METADATA."*