Learn Labs
7. Reliable Data Delivery

7.1 Kafka's reliability guarantees — the exact contract

The book frames this by analogy to ACID: "Those guarantees are the reason people trust relational databases with their most critical applications — they know exactly what the syst…

The book frames this by analogy to ACID: "Those guarantees are the reason people trust relational databases with their most critical applications — they know exactly what the system promises and how it will behave in different conditions. They understand the guarantees and can write safe applications by relying on them."

The four guarantees, verbatim

  1. Order within a partition. “If message B was written after message A, using the same producer in the same partition, then Kafka guarantees that the offset of B will be higher than A, and that consumers will read B after A.”
  2. The definition of committed. “Produced messages are considered ‘committed’ when they were written to the partition on all its in-sync replicas (but not necessarily flushed to disk).”
  3. Producers choose their acknowledgment point. “when the message was fully committed, when it was written to the leader, or when it was sent over the network.”
  4. Durability of committed messages. “Messages that are committed will not be lost as long as at least one replica remains alive.”
  5. Consumer visibility. “Consumers can only read messages that are committed.”

Read the fine print in each:

  • ① requires "the same producer in the same partition." Two producers writing to one partition have no relative ordering guarantee. Neither do two partitions.
  • ② "not necessarily flushed to disk" — the durability unit is replicas holding it, not bytes on platters (Ch. 6 §5.3).
  • ④ "as long as at least one replica remains alive" — lose all of them simultaneously and committed data is gone. This is why rack/AZ placement is a durability control, not a nicety.

"These basic guarantees can be used while building a reliable system, but in themselves, they don't make the system fully reliable."

The trade-off space, named explicitly:

reliable & consistentstorageavailabilityhigh throughputlow latencyhardware cost⇕ Every knob in this chapter buys one side by spending the other.
Figure 7.1.2The trade-off space, named explicitly

On this page