8.9 Self-test

Why are duplicates "an annoyance" in simple pipelines but a correctness disaster in aggregations?

Give the classic duplicate scenario. What is the crucial detail about whether the original write succeeded?

Why can no amount of producer-side retry logic solve that scenario? What must know the answer?

What four values uniquely identify a message for idempotence? How many messages does a broker track per partition?

Why does idempotence require max.in.flight.requests ≤ 5?

Duplicate rejections show up in which producer metric and which broker metric?

The broker reports "out of order sequence number." Why can you ignore it, and why should you absolutely not?

Does idempotence survive a producer restart? Explain precisely why or why not.

List the four layers of producer-state durability that let idempotence survive broker failure and restart.

A broker crashes and there are no messages in the retention window to rebuild state from. Why is this fine?

Give the two hard limitations of the idempotent producer, with a concrete example of each.

What exactly changes when you set enable.idempotence=true? What's the performance cost if acks=all is already set?

Which ordering problem from Ch. 3 does idempotence solve, and what does that let you keep that the folklore fix takes away?

Distinguish "transactions" from "exactly-once semantics." Which do Flink and Spark Streaming use?

Describe both problems transactions solve — crashes and zombies — and note why they need different mechanisms.

What's the insight that makes atomic offset-commit-plus-produce possible?

Contrast producer.id and transactional.id on origin, lifetime, and purpose.

How does epoch-based zombie fencing work here, and where else in Kafka have you seen the identical pattern?

isolation.level defaults to what, and why is that default fatal to exactly-once expectations?

What is the LSO? Draw what a read_committed consumer sees when a transaction is left open, and state the worst-case stall.

Name three things read_committed does not tell a consumer.

Your processing step sends an email. What does exactly-once guarantee about that email?

You need to write to a database and commit Kafka offsets atomically. What can't you do, and what should you do?

Describe both directions of the outbox pattern, and when you'd prefer the table-as-outbox variant.

Why can't Kafka preserve source-database transaction boundaries through a Kafka hop? Give both reasons.

MirrorMaker 2.0 can copy records exactly once. What can it not preserve, and why?

Compare Kafka's read_committed to JMS transaction visibility. Who enforces invisibility in each?

Describe the transaction deadlock pattern and why it's unavoidable once you enter it.

Producer A fails and is replaced by producer B; A returns as a zombie. Why isn't A fenced, and what are the two eras' fixes?

What does KIP-447 add, and what wasteful workaround does it eliminate?

What are the three configuration requirements for exactly-once with the raw API? Which one is easiest to get wrong?

What should you do on ProducerFencedException?

Give the four steps of the transaction algorithm. At which step are you "doomed to commit or abort eventually," and why does that matter?

What does beginTransaction() actually send to the broker? How does the coordinator learn which partitions are involved?

The transaction coordinator crashes after logging commit intent. What happens?

Do the producer-state memory arithmetic for 3 new producers/second over a week. What are the two mitigations?

Why does a larger transaction improve producer throughput, and what does it cost on the consumer side?

Transactions add latency but not throughput cost on the consumer. Explain both halves.

Previous: Chapter 7 — Reliable Data Delivery Next: Chapter 9 — Building Data Pipelines