Learn Labs
2. Installing Kafka

2.0 The shape of a Kafka deployment

Everything else is tuning.

zookeeper.connectzoo1myid = 1zoo2myid = 2zoo3myid = 3broker.id = 1log.dirs · broker.rackbroker.id = 2log.dirs · broker.rackbroker.id = 3log.dirs · broker.rackproducersconsumersZOOKEEPER ENSEMBLE — odd member countports: 2181 client · 2888 peer · 3888 leader-electionlisteners: PLAINTEXT://:9092,SSL://:9091

zookeeper.connect = zoo1:2181,zoo2:2181,zoo3:2181/kafka-cluster-a — the trailing chroot path is what lets the ensemble “be shared with other applications, including other Kafka clusters, without conflict.” If the chroot path doesn’t exist, the broker creates it at startup.

  • · Every broker has the same zookeeper.connect — same ensemble and same path.
  • · Every broker has a unique broker.id. If two brokers try to join with the same ID, the second logs an error and fails to start.
  • · Those are the only two requirements to form a cluster. Everything else is tuning.
Figure 2.0.1The shape of a Kafka deployment

Two hard requirements to form a cluster — that's all:

  1. Every broker has the same zookeeper.connect (same ensemble + same path).
  2. Every broker has a unique broker.id. If two brokers try to join with the same ID, the second logs an error and fails to start.

Everything else is tuning.

2. Installing Kafka

Chapter 2 of Kafka: The Definitive Guide — 12 sections.

2.1 What problem does this chapter solve?

The default server.properties shipped with Kafka is "sufficient to run a standalone server as a proof of concept, but most likely will not be sufficient for large installations."