11.8 Securing ZooKeeper

"ZooKeeper stores Kafka metadata that is critical for maintaining the availability of Kafka clusters, and hence it is VITAL to secure ZooKeeper in addition to securing Kafka."

8.1 SASL

"SASL configuration for ZooKeeper is provided using the Java system property java.security.auth.login.config" pointing at a JAAS file.

ZooKeeper server JAAS (Server section):

Server {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true storeKey=true
  keyTab="/path/to/zk.keytab"
  principal="zookeeper/zk1.example.com@EXAMPLE.COM";
};

ZooKeeper server config:

authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true

⚠️ BROKER PRINCIPAL — why those two flags matter

"By default, ZooKeeper uses the FULL Kerberos principal, e.g. kafka/broker1.example.com@EXAMPLE.COM, as the client identity. When ACLs are enabled for ZooKeeper authorization, ZooKeeper servers SHOULD be configured with kerberos.removeHostFromPrincipal=true and kerberos.removeRealmFromPrincipal=true TO ENSURE THAT ALL BROKERS HAVE THE SAME PRINCIPAL."

Kafka broker's ZooKeeper client JAAS (Client section):

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true storeKey=true
  keyTab="/path/to/broker1.keytab"
  principal="kafka/broker1.example.com@EXAMPLE.COM";
};

8.2 SSL

⚠️ The key difference from Kafka: "Like Kafka, SSL may be configured to enable client authentication, BUT UNLIKE KAFKA, connections with BOTH SASL AND SSL client authentication AUTHENTICATE USING BOTH PROTOCOLS AND ASSOCIATE MULTIPLE PRINCIPALS WITH THE CONNECTION. ZooKeeper authorizer grants access to a resource IF ANY OF THE PRINCIPALS associated with the connection have access."

ZooKeeper server:

secureClientPort=2181
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
ssl.keyStore.location=/path/to/zk.ks.p12
ssl.keyStore.password=zk-ks-password
ssl.keyStore.type=PKCS12
ssl.trustStore.location=/path/to/zk.ts.p12
ssl.trustStore.password=zk-ts-password
ssl.trustStore.type=PKCS12

Kafka broker → ZooKeeper:

zookeeper.ssl.client.enable=true
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.keystore.location=/path/to/zkclient.ks.p12
zookeeper.ssl.keystore.password=zkclient-ks-password
zookeeper.ssl.keystore.type=PKCS12
zookeeper.ssl.truststore.location=/path/to/zkclient.ts.p12
zookeeper.ssl.truststore.password=zkclient-ts-password
zookeeper.ssl.truststore.type=PKCS12

8.3 ZooKeeper authorization

zookeeper.set.acl=true    # on the BROKERS

"the broker sets ACLs for ZooKeeper nodes WHEN CREATING THE NODE. By default, metadata nodes are READABLE BY EVERYONE but MODIFIABLE ONLY BY BROKERS. Additional ACLs may be added if required for internal admin users who may need to update metadata directly. SENSITIVE PATHS, LIKE NODES CONTAINING SCRAM CREDENTIALS, ARE NOT WORLD-READABLE BY DEFAULT."