Learn Labs
3. Kafka Producers: Writing Messages to Kafka

3.6 ⚠️ ORDERING GUARANTEES — the reordering bug

"Apache Kafka preserves the order of messages within a partition. If messages are sent from the producer in a specific order, the broker will write them to a partition in that order and all consumers will read them in that order. For some use cases, order is very important. There is a big difference between depositing $100 in an account and later withdrawing it, and the other way around!"

How retries silently reorder your data

retries > 0 AND max.in.flight.requests.per.connection > 1

producerbrokerpartition logt0 · Batch At1 · Batch Bin.flight > 1t2 · A FAILS to writet3 · B writtent4 · RETRY At5 · A written

LOG ORDER: [ … B, A … ] — ORDER REVERSED. Silently.

Figure 3.6.1How retries silently reorder your data

The bind, and the resolution

≥ 2 in-flight requestsfor performancea high retry allowancefor reliabilitytogether they break orderingcorrectnessSOLUTION: enable.idempotence = true
  • · guarantees message ordering with up to FIVE in-flight requests
  • · AND guarantees that retries will not introduce duplicates
Figure 3.6.2The bind, and the resolution

(Ch. 8 covers the idempotent producer in depth.)

Do not use the old folklore fix (max.in.flight.requests.per.connection=1) — it costs you throughput and is strictly worse than idempotence.

3.5 Message delivery time — the timeout configs and how they interact

How long the producer may block when calling send() and when explicitly requesting metadata via partitionsFor().

3.7 Remaining configs

Controls the size of a produce request. It caps both:

On this page

How retries silently reorder your dataThe bind, and the resolution