Learn Labs
5. Managing Apache Kafka Programmatically (AdminClient)

5.3 Lifecycle: create, configure, close

Introduced in Kafka 2.1.0. Default behavior: "Kafka validates, resolves, and creates connections based on the hostname provided in the bootstrap server configuration (and later in…

Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
AdminClient admin = AdminClient.create(props);
// TODO: Do something useful with AdminClient
admin.close(Duration.ofSeconds(30));

Only mandatory config: the cluster URI — a comma-separated broker list. "As usual, in production environments, you want to specify at least three brokers just in case one is currently unavailable."

(Note the escalation: Ch. 3 said "at least two" for producers; here it's "at least three." Admin operations are rarer and more critical, so failing to bootstrap is worse.)

close() semantics — read carefully

timeout expiresadmin.close(timeout)no more callscan't send any more requestswaits for responsesuntil the timeout expiresafter the timeoutaborts ALL ongoing operations
  • Once you call close, you can't call any other methods or send any more requests.
  • But the client will wait for responses until the timeout expires — there could still be operations in progress.
  • After the timeout: aborts all ongoing operations with a timeout exception, and releases all resources.
  • admin.close() with no timeout waits as long as it takes for all ongoing operations to complete.
Figure 5.3.1close() semantics — read carefully

"AdminClient is much simpler [than producer/consumer], and there is not much to configure."

3.1 client.dns.lookup — two distinct problems, two different values

Introduced in Kafka 2.1.0. Default behavior: "Kafka validates, resolves, and creates connections based on the hostname provided in the bootstrap server configuration (and later in the names returned by the brokers as specified in advertised.listeners)."

"This simple model works most of the time but fails to cover two important use cases." They are mutually exclusive scenarios.

Problem A: DNS alias + SASL → authentication failure
authenticatesnames don't matchclientbootstrap = the aliasall-brokers.hostname.comONE alias instead of a full listbroker2.hostname.comthe server principalSASL REFUSES to authenticateconnection fails

Maintaining a full bootstrap list “can easily become challenging to maintain,” so you create one alias — “you don't actually care which broker gets the initial connection.” Very convenient, unless you use SASL: the client authenticates all-brokers.hostname.com while the server principal is broker2.hostname.com, and from SASL's perspective a mismatched broker certificate could be a man-in-the-middle attack.

Figure 5.3.2Problem A: DNS alias + SASL → authentication failure

Fix:

client.dns.lookup=resolve_canonical_bootstrap_servers_only

"the client will 'expend' the DNS alias, and the result will be the same as if you included all the broker names the DNS alias connects to as brokers in the original bootstrap list."

Problem B: one DNS name → multiple IPs (load balancers) → false unavailability
the FIRST IP resolvedbroker1.hostname.comone DNS nameIP1unavailableIP2load balancerIP3load balancerthe SAME brokerFULLY AVAILABLE

Brokers behind a proxy or load balancer are “especially common if you use Kubernetes, where load balancers are necessary to allow connections from outside the Kubernetes cluster,” and you don't want the LB to be a single point of failure — so one name resolves to several IPs, all routing to the same broker, and these IPs change over time. The default client “will just try to connect to the first IP that the hostname resolves,” so if that IP becomes unavailable the client fails to connect even though the broker is fully available: you built an HA load-balancing layer and got zero benefit from it.

Figure 5.3.3Problem B: one DNS name → multiple IPs (load balancers) → false unavailability

Fix:

client.dns.lookup=use_all_dns_ips

"highly recommended ... to make sure the client doesn't miss out on the benefits of a highly available load balancing layer."

Summary:

ScenarioValueSymptom without it
DNS alias for bootstrap + SASLresolve_canonical_bootstrap_servers_onlySASL auth failure / connection refused
One DNS name → multiple IPs (LBs, Kubernetes)use_all_dns_ipsClient can't connect even though brokers are healthy

3.2 request.timeout.ms (default 120 seconds)

"limits the time your application can spend waiting for AdminClient to respond. This includes the time spent on retrying if the client receives a retriable error."

Why the default is so long: "quite long, but some AdminClient operations, especially consumer group management commands, can take a while to respond."

Per-call override via Options — and the book gives a very practical pattern:

"If an AdminClient operation is on the critical path for your application, you may want to use a lower timeout and handle a lack of timely response from Kafka in a different way. A common example is that services try to validate the existence of specific topics when they first start, but if Kafka takes longer than 30 seconds to respond, you may want to continue starting the server and validate the existence of topics later (or skip this validation entirely)."

startup topic validationresponds in <30svalidate, proceedtimes outSTART ANYWAY

On the timeout branch you validate later, or skip the validation entirely — don't let Kafka slowness become your startup failure.

Figure 5.3.43.2 request.timeout.ms (default 120 seconds)

This is a genuinely good availability pattern: don't couple your service's startup liveness to another system's responsiveness.


On this page