14. Stream Processing
14.0 Further reading — the chapter's own bibliography
"This chapter is intended as just a quick introduction to the large and fascinating world of stream processing and Kafka Streams. THERE ARE ENTIRE BOOKS WRITTEN ON THESE SUBJECTS."
Concepts — "the basic concepts of stream processing from a DATA ARCHITECTURE perspective":
| Book | Author(s) | Publisher | What it's for |
|---|---|---|---|
| Making Sense of Stream Processing | Martin Kleppmann | O'Reilly | "discusses the benefits of RETHINKING APPLICATIONS AS STREAM PROCESSING APPLICATIONS and how to REORIENT DATA ARCHITECTURES around the idea of event streams" |
| Streaming Systems | Tyler Akidau, Slava Chernyak, Reuven Lax | O'Reilly | "a great GENERAL INTRODUCTION to the topic of stream processing and some of the basic ideas in the space" |
| Flow Architectures | James Urquhart | O'Reilly | "targeted at CTOs and discusses the IMPLICATIONS of stream processing TO THE BUSINESS" |
Frameworks — "specific details of specific frameworks":
| Book | Author(s) | Publisher |
|---|---|---|
| Mastering Kafka Streams and ksqlDB | Mitch Seymour | O'Reilly |
| Kafka Streams in Action | William P. Bejeck Jr. | Manning |
| Event Streaming with Kafka Streams and ksqlDB | William P. Bejeck Jr. | Manning |
| Stream Processing with Apache Flink | Fabian Hueske, Vasiliki Kalavri | O'Reilly |
| Stream Processing with Apache Spark | Gerard Maas, Francois Garillot | O'Reilly |
Referenced elsewhere in the chapter:
- "There Is No Now" — Justin Sheehy — "an excellent paper" on how complex time gets in distributed systems (§2.2)
- "Crossing the Streams" — Kafka Summit 2020 talk, plus a more in-depth blog post — on foreign-key joins (§3.5)
- "Beyond the DSL" — presentation — "a great introduction" to the low-level Processor API (§4)
- "Testing Kafka Streams — A Deep Dive" — blog post — deeper explanations and detailed code examples of topologies and tests (§5.3)
- A blog post and a Kafka Summit talk on Kafka Streams scalability and high availability (§5.6)
- The Kafka Streams developer guide — for the low-level Processor API (§4)
| Which one to reach for | The book |
|---|---|
| “should we rethink our architecture around event streams?” | Kleppmann, Making Sense of Stream Processing |
| “I need the concepts — time, windows, watermarks, correctness” | Akidau et al., Streaming Systems — the canonical text on the event-time/processing-time problem this chapter introduces in §2.2 |
| “I need to sell this internally” | Urquhart, Flow Architectures |
| “I'm building on Kafka Streams” | Seymour (O'Reilly) or Bejeck (Manning) |
| “I picked Flink for low-millisecond latency” (App. B) | Hueske & Kalavri, Stream Processing with Apache Flink |
| “we already run Spark for batch” (App. B) | Maas & Garillot, Stream Processing with Apache Spark |
⚠️ Version caveat: "Kafka Streams is still an evolving framework. EVERY MAJOR RELEASE DEPRECATES APIs AND MODIFIES SEMANTICS. This chapter documents APIs and semantics as of Apache Kafka 2.8. We avoided using any API planned for deprecation in 3.0, but our discussion of JOIN SEMANTICS and TIMESTAMP HANDLING does NOT include any of the changes planned for release 3.0."