3.7 Remaining configs

max.request.size

Controls the size of a produce request. It caps both:

1. the size of the largest message that can be sent, and
2. the number of messages the producer can send in one request.

"With a default maximum request size of 1 MB, the largest message you can send is 1 MB, or the producer can batch 1,024 messages of size 1 KB each into one request."

Coordinate with the broker:

"the broker has its own limit on the size of the largest message it will accept (message.max.bytes). It is usually a good idea to have these configurations match, so the producer will not attempt to send messages of a size that will be rejected by the broker."

receive.buffer.bytes and send.buffer.bytes

TCP send/receive buffer sizes used by the sockets. -1 → use OS defaults.

"It is a good idea to increase these when producers or consumers communicate with brokers in a different datacenter, because those network links typically have higher latency and lower bandwidth."

enable.idempotence

Kafka has supported exactly-once semantics since version 0.11. The idempotent producer is "a simple and highly beneficial part of it."

The duplicate scenario — trace it carefully:

acks=all with a generous delivery.timeout.ms — “maximize reliability” — means each message is written to Kafka at least once, which sometimes means more than once.

The retry goes to the new leader, which already has a copy of the record. Nothing failed: the data was written and replicated correctly — only the acknowledgment was lost.

Note what makes this so nasty: nothing failed. The data was written and replicated correctly. Only the acknowledgment was lost. Retry-on-timeout is fundamentally unable to distinguish "write failed" from "ack lost."

The fix:

"When the idempotent producer is enabled, the producer will attach a sequence number to each record it sends. If the broker receives records with the same sequence number, it will reject the second copy and the producer will receive the harmless DuplicateSequenceException."

NOTE — required preconditions:

  max.in.flight.requests.per.connection  ≤ 5
  retries                                > 0
  acks                                   = all

"If incompatible values are set, a ConfigException will be thrown."

---