11. Securing Kafka
11.1 The five security procedures, and the reference data flow
Note that availability is a security concern here, not just an ops concern — and that it explicitly includes ZooKeeper.
| Procedure | What it establishes |
|---|---|
| Authentication | “establishes your identity and determines who you are” |
| Authorization | “determines what you are allowed to do” |
| Encryption | “protects your data from eavesdropping and tampering” |
| Auditing | “tracks what you have done or have attempted to do” |
| Quotas | “control how much resources you can utilize” |
The reference flow used throughout the chapter
Every guarantee in the chapter is stated against this flow: the broker must know the message really came from Alice, Alice's client must verify it is talking to the real broker, and the leader must check that Alice may write to customerOrders and that Bob may read it — “if Bob's consumer uses group management, the broker should also verify that Bob has access to the consumer group.” Bob consumes customerOrders at ⑤, and the internal metrics application is the sixth participant in the flow.
The seven guarantees a secure deployment must provide
| Guarantee | What it means in the flow |
|---|---|
| Client authenticity | "the broker should authenticate the client to ensure that the message is really coming from Alice" |
| Server authenticity | "Alice's client should verify that the connection is to the REAL broker" |
| Data privacy | "ALL connections where the message flows, as well as ALL DISKS where messages are stored, should be encrypted or physically secured" |
| Data integrity | "Message digests should be included for data transmitted over insecure networks to detect tampering" |
| Access control | Leader verifies Alice may write to customerOrders; verifies Bob may read; "If Bob's consumer uses group management, the broker should ALSO verify that Bob has access to the CONSUMER GROUP" |
| Auditability | "An audit trail that shows all operations performed by brokers, Alice, Bob, and other clients" |
| Availability | "quotas and limits to avoid some users hogging all the available bandwidth or overwhelming the broker with DoS attacks. ZooKeeper should be locked down since broker availability is dependent on ZooKeeper availability and the integrity of metadata stored in ZooKeeper" |
Note that availability is a security concern here, not just an ops concern — and that it explicitly includes ZooKeeper.