12.3 Dynamic configuration changes — `kafka-configs.sh`
controller_mutations_rate is the guard against the controller-overload problem from §1.7 — someone scripting mass topic creation/deletion.
"There is a plethora of configurations for topics, clients, brokers, and more that can be updated DYNAMICALLY DURING RUNTIME without having to shut down or redeploy a cluster."
Four entity types: topics, brokers, users, clients.
"New dynamic configs are being added constantly with each release, so it is good to ensure you have the same version of this tool that matches the version of Kafka you are running."
💡 "For ease of setting up these configs consistently via automation, the
--add-config-fileargument can be used with a preformatted file of all the configs you want to manage and update."
3.1 Topic configuration overrides
kafka-configs.sh --bootstrap-server localhost:9092 \
--alter --entity-type topics --entity-name my-topic \
--add-config retention.ms=3600000
# Updated config for topic: "my-topic".The point: "we can override the cluster-level defaults for individual topics to accommodate DIFFERENT USE CASES WITHIN A SINGLE CLUSTER." (Ch. 7's bank example: strict defaults, relaxed complaints topic.)
Valid topic config keys (Table 12-2), grouped by purpose:
| Key | Meaning |
|---|---|
| Retention & cleanup | |
cleanup.policy | compact → “only the most recent message with a given key is retained (log compacted)” |
retention.ms | how long to retain, in ms |
retention.bytes | how much to retain, in bytes |
segment.bytes | bytes written to a single log segment |
segment.ms | how frequently the log segment should be rotated |
segment.jitter.ms | “randomized and added to segment.ms when rolling” ◄ THE FIX for Ch. 2’s synchronized-roll storm |
segment.index.bytes | max size of a single log segment index |
file.delete.delay.ms | wait before deleting log segments/indices from disk |
preallocate | preallocate log segments when a new one is rolled |
| Compaction | |
delete.retention.ms | “how long deleted TOMBSTONES will be retained. Only valid for log compacted topics” ◄ THE GDPR knob from Ch. 6 §7.4 |
min.cleanable.dirty.ratio | “how frequently the log compactor will attempt to compact... ratio of uncompacted segments to total” |
min.compaction.lag.ms | minimum time a message remains uncompacted |
max.compaction.lag.ms | “Maximum time limit a message won’t be eligible for compaction” ◄ the other GDPR knob |
| Durability | |
min.insync.replicas | “minimum replicas that must be in sync for a partition to be considered available” |
unclean.leader.election.enable | false → no unclean elections for this topic |
flush.messages | messages received before forcing a flush to disk |
flush.ms | time before forcing a flush to disk |
| Messages & format | |
compression.type | codec used by the BROKER when writing batches |
max.message.bytes | max size of a single message |
message.format.version | format version used when writing to disk |
message.downconversion.enable | “Allows the message format version to be down-converted to the previous version if enabled WITH SOME OVERHEAD” ◄ Ch. 6 §6.5’s CPU landmine |
message.timestamp.type | CreateTime (client) | LogAppendTime (broker) |
message.timestamp.difference.max.ms | max client/broker timestamp skew — “only valid if message.timestamp.type = CreateTime” |
index.interval.bytes | bytes producible between index entries |
| Replication throttling | |
leader.replication.throttled.replicas | throttled BY THE LEADER |
follower.replication.throttled.replicas | throttled BY THE FOLLOWER |
Note message.downconversion.enable — you can disable down-conversion per topic, forcing old clients to fail loudly instead of silently burning broker CPU (Ch. 6 §6.5).
3.2 Client and user overrides — all quotas
"For Kafka clients and users, there are only a few configurations that can be overridden, which are all essentially types of QUOTAS."
| Config key | Description |
|---|---|
consumer_bytes_rate | "bytes a single client ID is allowed to consume from a single broker in one second" |
producer_bytes_rate | "bytes a single client ID is allowed to produce to a single broker in one second" |
controller_mutations_rate | "The rate at which mutations are accepted for the create topics request, the create partitions request, and the delete topics request. The rate is accumulated by the NUMBER OF PARTITIONS created or deleted." |
request_percentage | "The percentage per quota window (out of a total of (num.io.threads + num.network.threads) × 100%) for requests from the user or client" |
controller_mutations_rate is the guard against the controller-overload problem from §1.7 — someone scripting mass topic creation/deletion.
⚠️ Quotas are PER-BROKER — the balance dependency
"Because throttling occurs on a PER-BROKER basis, EVEN BALANCE OF LEADERSHIP of partitions across a cluster becomes PARTICULARLY IMPORTANT to enforce this properly."
5 brokers, producer quota = 10 MBps for a client:
- Balanced leadership
- 10 MBps × 5 brokers = 50 MBps Total for that client
- All leadership on broker 1
- 10 MBps Total
► The Same quota config yields a 5× different effective limit depending on leadership balance. Your quota is only as meaningful as your balance.
CLIENT ID vs CONSUMER GROUP
*"The client ID is NOT necessarily the same as the consumer group name. Consumers can set their own client ID, and you may have many consumers that are in DIFFERENT GROUPS that specify the SAME client ID.
💡 "It is considered a best practice to set the client ID for each consumer group to something unique that identifies that group. This allows a single consumer group to SHARE A QUOTA, and it makes it easier to identify in logs what group is responsible for requests."
Combining user and client in one command:
kafka-configs.sh --bootstrap-server localhost:9092 \
--alter --add-config "controller_mutations_rate=10" \
--entity-type clients --entity-name <client ID> \
--entity-type users --entity-name <user ID>3.3 Broker overrides
"More than 80 overrides can be altered with
kafka-configs.shfor brokers." Three called out specifically:
| Config | Purpose |
|---|---|
min.insync.replicas | "Adjusts the minimum number of replicas that need to acknowledge a write for a produce request to be successful when producers have set acks to all (or –1)" |
unclean.leader.election.enable | "Allows replicas to be elected as leader even if it results in data loss. Useful when it is permissible to have some lossy data, or to turn on FOR SHORT TIMES to UNSTICK a Kafka cluster if unrecoverable data loss cannot be avoided." |
max.connections | "maximum connections allowed to a broker at any time. We can also use max.connections.per.ip and max.connections.per.ip.overrides for more fine-tuned throttling." |
💡 The fact that unclean.leader.election.enable is dynamically changeable is important — Ch. 7 §3.2 described it as requiring a restart, but as a dynamic broker config you can flip it on, recover, and flip it back without bouncing the cluster.
3.4 Describing and removing overrides
kafka-configs.sh --bootstrap-server localhost:9092 \
--describe --entity-type topics --entity-name my-topic
# Configs for topics:my-topic are
# retention.ms=3600000⚠️ TOPIC OVERRIDES ONLY
"The configuration description will ONLY SHOW OVERRIDES — it does NOT include the cluster DEFAULT configurations. THERE IS NOT A WAY TO DYNAMICALLY DISCOVER THE CONFIGURATION OF THE BROKERS THEMSELVES. This means that when using this tool to discover topic or client settings in AUTOMATION, THE USER MUST HAVE SEPARATE KNOWLEDGE OF THE CLUSTER DEFAULT CONFIGURATION."
(Contrast Ch. 5 §5: the AdminClient's describeConfigs does return defaults with isDefault() — so the programmatic API is strictly more capable here than the CLI.)
Removing an override reverts to the cluster default:
kafka-configs.sh --bootstrap-server localhost:9092 \
--alter --entity-type topics --entity-name my-topic \
--delete-config retention.ms