4.4 Creating a consumer

Three mandatory + one near-mandatory property

Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("group.id", "CountryCounter");
props.put("key.deserializer",
    "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
    "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer =
    new KafkaConsumer<String, String>(props);

Subscribing

// explicit list
consumer.subscribe(Collections.singletonList("customerCountries"));

// regular expression
consumer.subscribe(Pattern.compile("test.*"));

Regex subscription behavior: "if someone creates a new topic with a name that matches, a rebalance will happen almost immediately and the consumers will start consuming from the new topic." Most commonly used in "applications that replicate data between Kafka and another system, or streams processing applications."

⚠️ WARNING — regex subscription is a client-side scan with real cost

If your cluster has a large number of partitions — perhaps 30,000 or more — be aware that the filtering of topics for the subscription is done on the client side.

What actually happens:

 consumer requests THE LIST OF ALL TOPICS AND THEIR PARTITIONS
 from the broker at REGULAR INTERVALS
   → client uses this list to detect new matching topics
   → subscribes to them

Consequences when the topic list is large and there are many consumers:

• "the size of the list of topics and partitions is significant"
• "the regular expression subscription has significant overhead on the broker, client, and network"
• "There are cases where the bandwidth used by the topic metadata is larger than the bandwidth used to send data."

And a security consequence people miss: "in order to subscribe with a regular expression, the client needs permissions to describe all topics in the cluster — that is, a full describe grant on the entire cluster."

That last point often blocks regex subscriptions outright in locked-down multi-tenant clusters — you cannot grant least-privilege topic access and use regex subscription.