12.1 Topic operations — `kafka-topics.sh`

"allows you to create, modify, delete, and list information about topics. While some topic CONFIGURATIONS are possible through this command, they have been DEPRECATED, and it is recommended to use the more robust method of using the kafka-config.sh tool for configuration changes."

1.1 Creating a topic

Three required arguments — "These arguments must be provided even though some of them may have broker-level defaults configured already."

kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic my-topic --replication-factor 2 --partitions 8
# Created topic "my-topic".

Rack awareness: "if the cluster is set up for rack-aware replica assignment, the replicas for each partition will be in separate racks. If rack-aware assignment is not desired, specify --disable-rack-aware."

⚠️ GOOD TOPIC NAMING PRACTICES

"Topic names may contain alphanumeric characters, underscores, dashes, and periods; however:

① DO NOT USE PERIODS. "Internal metrics inside of Kafka CONVERT PERIOD CHARACTERS TO UNDERSCORE CHARACTERS (e.g., 'topic.1' becomes 'topic_1' in metrics calculations), which can result in CONFLICTS in topic names."

② DO NOT START WITH A DOUBLE UNDERSCORE. "By convention, topics internal to Kafka operations are created with a double underscore naming convention (like the __consumer_offsets topic)... it is not recommended to have topic names that begin with the double underscore naming convention to prevent confusion."

--if-exists / --if-not-exists

• --if-not-exists with --create: "you may want to use [it] in automation... that will not return an error if the topic already exists." ✅
• --if-exists with --alter: "using it is NOT RECOMMENDED. Using this argument will cause the command to not return an error if the topic being changed DOES NOT EXIST. THIS CAN MASK PROBLEMS where a topic does not exist that should have been created." ❌

1.2 Listing topics

kafka-topics.sh --bootstrap-server localhost:9092 --list
# __consumer_offsets
# my-topic
# other-topic

"formatted with one topic per line, in NO PARTICULAR ORDER." Add --exclude-internal to "remove all topics from the list that begin with the double underscore."

1.3 Describing topics

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my-topic

Topic: my-topic PartitionCount: 8 ReplicationFactor: 2 Configs: segment.bytes=1073741824
  Topic: my-topic Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0
  Topic: my-topic Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1
  ...

Output includes "the partition count, topic CONFIGURATION OVERRIDES, and a listing of each partition with its replica assignments."

(Recall Ch. 6 §4.4: the first replica in the Replicas list is the preferred leader — here p0 prefers broker 1, p1 prefers broker 0. That alternation is what balanced leadership looks like.)

💡 1.4 The diagnostic filters — the most useful part of the whole chapter

"These can be helpful for diagnosing cluster issues more easily. For these commands we generally do NOT specify the --topic argument because the intention is to find all topics or partitions that match the criteria. These options will NOT work with the --list command."

The escalating severity ladder — memorize this:

Worked example — min-ISR 1, RF 2, host 0 up, host 1 down for maintenance:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --at-min-isr-partitions
  Topic: my-topic Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0
  ...

Note Replicas: 0,1 but Isr: 0 — exactly at min-ISR, zero redundancy remaining.

1.5 Adding partitions

Why: "the most common reason is to horizontally scale a topic across more brokers by decreasing the throughput for a single partition. Topics may also be increased if a consumer needs to expand to run more copies in a single consumer group since a partition can only be consumed by a single member in the group."

kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter --topic my-topic --partitions 16

⚠️ ADJUSTING KEYED TOPICS

"Topics that are produced with keyed messages can be VERY DIFFICULT to add partitions to from a consumer's point of view. This is because THE MAPPING OF KEYS TO PARTITIONS WILL CHANGE when the number of partitions is changed. For this reason, it is advisable to SET THE NUMBER OF PARTITIONS ONCE, WHEN THE TOPIC IS CREATED, AND AVOID RESIZING."

(Third appearance of this warning — Ch. 3 §9.4, Ch. 5 §7.2, and here. Kafka's authors really mean it.)

1.6 ⚠️ Reducing partitions is impossible

"It is NOT POSSIBLE to reduce the number of partitions. Deleting a partition would cause part of the data in that topic to be deleted as well, which would be INCONSISTENT from a client point of view. In addition, trying to redistribute the data to the remaining partitions would be difficult and result in OUT-OF-ORDER messages."

The two workarounds:

1. Delete the topic and Re-create it
2. If deletion is not possible: Create a new version of the topic and Move all produce traffic to it (e.g. “my-topic-v2”)

1.7 Deleting a topic

Why bother deleting empty topics:

"Even a topic with no messages uses cluster resources such as disk space, open filehandles, and memory. THE CONTROLLER ALSO HAS JUNK METADATA that it must retain knowledge of, WHICH CAN HINDER PERFORMANCE AT LARGE SCALE."

kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic my-topic
# Note: This will have no impact if delete.topic.enable is not set to true.

Prerequisite: delete.topic.enable=true on the brokers. "If it's set to false, then the request to delete the topic will be IGNORED and will not succeed."

It's asynchronous, and there's a rate limit you should respect:

“running this command will Mark a topic for deletion, but the deletion may Not happen immediately, depending on the amount of data and cleanup needed. The Controller will notify the brokers of the pending deletion As soon as possible (after existing controller tasks complete), and the brokers will then Invalidate the metadata and Delete the files from disk.”

⚠ “It is highly recommended that operators not delete more than one or two topics at A time, and give those ample time to complete before deleting other topics, Due to limitations in the way the controller executes these operations.”

⚠️ DATA LOSS AHEAD

"Deleting a topic will also delete all its messages. THIS IS NOT A REVERSIBLE OPERATION. Make sure it is executed carefully."

And there is no success feedback: "You will notice there is NO VISIBLE FEEDBACK that the topic deletion was completed successfully or not. Verify that deletion was successful by running the --list or --describe options."