11.5 Encryption
That's a real operational cost of combining end-to-end encryption with compacted topics: key rotation becomes a maintenance window.
Three layers, three different threats
| Layer | Mechanism | Threat addressed |
|---|---|---|
| ① In transit | TLS via the SSL / SASL_SSL security protocols. “TLS cipher suites can be restricted to strengthen security and adhere to security requirements like FIPS.” | Network eavesdropping, tampering. |
| ② At rest | “physical storage can be encrypted using whole disk encryption or volume encryption.” | “sensitive data cannot be retrieved even by users with physical access to the disk”; “avoid security breaches even if the disk is stolen”. |
| ③ End-to-end | Encryption in the serializer/deserializer with a key from a KMS — the threat ① and ② do not cover. | The platform administrator: “additional protection may be required to avoid granting automatic data access to platform administrators. Unencrypted data present in broker memory may appear in heap dumps, and administrators with direct access to the disk will be able to access these, as well as Kafka logs containing potentially sensitive data.” “To comply with regulatory requirements, especially in cloud deployments, it is necessary to guarantee that confidential data cannot be accessed by platform administrators or cloud providers by any means.” |
Heap dumps are the underrated threat. TLS + disk encryption still leaves plaintext in broker RAM.
5.1 End-to-end encryption
Where it hooks in: "Serializers and deserializers can be integrated with an encryption library to perform encryption of the message during serialization, and decryption during deserialization." (Ch. 3/Ch. 4.)
The broker stores the encrypted message in its partition logs. ⚠ “Brokers do not require access to the encryption key and never see the unencrypted contents, making this approach safe to use in cloud environments.”
Details:
- “typically performed using symmetric encryption algorithms like AES”
- “A shared encryption key stored in a key management system (KMS)”
- “Encryption parameters required to decrypt the message may be stored in message headers or in the message payload if older consumers without header support need access.”
- “A digital signature may also be included in message headers to verify message integrity.”
Key rotation — and the compacted-topic problem
"Periodic key rotation is recommended... since frequent rotation limits the number of compromised messages in case of a breach and also protects against brute-force attacks. CONSUMPTION MUST BE SUPPORTED WITH BOTH OLD AND NEW KEYS DURING THE RETENTION PERIOD of messages encrypted with the old key. Many KMS systems support graceful key rotation out of the box for symmetric encryption without requiring special handling in Kafka clients."
⚠️ "For COMPACTED TOPICS, messages encrypted with old keys MAY BE RETAINED FOR A LONG TIME, and it may be necessary to RE-ENCRYPT OLD MESSAGES. To avoid interference with newer messages, PRODUCERS AND CONSUMERS MUST BE OFFLINE DURING THIS PROCESS."
That's a real operational cost of combining end-to-end encryption with compacted topics: key rotation becomes a maintenance window.
⚠️ COMPRESSION OF ENCRYPTED MESSAGES
*"Compressing messages AFTER encryption is UNLIKELY TO PROVIDE ANY BENEFIT in terms of space reduction compared to compressing prior to encryption. Serializers may be configured to perform compression BEFORE encrypting, or applications may compress prior to producing. In either case, IT IS BETTER TO DISABLE COMPRESSION IN KAFKA since it adds overhead without providing any additional benefit.
For messages transmitted over an insecure transport layer, KNOWN SECURITY EXPLOITS OF COMPRESSED ENCRYPTED MESSAGES must also be taken into account."*
(Encrypted data is high-entropy → incompressible. And compression-plus-encryption oracles like CRIME/BREACH are the "known exploits" being referenced.)
⚠️ Encrypting message keys — the hash-equivalence problem
*"In many environments, especially when TLS is used, message keys do not require encryption since they typically do not contain sensitive data. But in some cases, clear-text keys may not comply with regulatory requirements.
Since message keys are used for PARTITIONING and COMPACTION, transformation of keys MUST PRESERVE THE REQUIRED HASH EQUIVALENCE to ensure that a key RETAINS THE SAME HASH VALUE even if encryption parameters are altered.
One approach: store a SECURE HASH of the original key AS the message key, and store the ENCRYPTED message key in the message PAYLOAD or in a HEADER.
Since Kafka serializes message key and value INDEPENDENTLY, a PRODUCER INTERCEPTOR can be used to perform this transformation."*