4.1 What problem do consumer groups solve?

The starting point: an application reads from a topic, runs validations, writes results to another data store. One consumer object, one subscription. Works fine.

Then: "what if the rate at which producers write messages to the topic exceeds the rate at which your application can validate them? If you are limited to a single consumer reading and processing the data, your application may fall further and further behind, unable to keep up with the rate of incoming messages."

Why this is the normal case, not an edge case:

"It is common for Kafka consumers to do high-latency operations such as write to a database or a time-consuming computation on the data. In these cases, a single consumer can't possibly keep up with the rate data flows into a topic."

The solution: "Just like multiple producers can write to the same topic, we need to allow multiple consumers to read from the same topic, splitting the data among them."

The scaling ladder — memorize these four states

"The main way we scale data consumption from a Kafka topic is by adding more consumers to a consumer group. ... This is a good reason to create topics with a large number of partitions — it allows adding more consumers when the load increases. Keep in mind that there is no point in adding more consumers than you have partitions in a topic — some of the consumers will just be idle."

This closes the loop with Ch. 2: partition count is your permanent ceiling on consumer parallelism, and partitions can only be added (which breaks keyed routing, per Ch. 3). Hence: size partitions for future consumer parallelism, up front.

Multiple groups = multiple independent readers

"One of the main design goals in Kafka was to make the data produced to Kafka topics available for many use cases throughout the organization. ... To make sure an application gets all the messages in a topic, ensure the application has its own consumer group. Unlike many traditional messaging systems, Kafka scales to a large number of consumers and consumer groups without reducing performance."

The rule, stated once

• NEW consumer GROUP → for each APPLICATION that needs ALL messages.
• NEW consumer in a group → to SCALE reading/processing within one app (each additional member gets a SUBSET).