14. Stream Processing
14.7 How to choose a stream processing framework
Note that two of the four answers are "you may not want stream processing at all." That's an unusually honest framing for a chapter selling a stream processing library.
7.1 Four application types → four different answers
| Application type | What it is | What to look for |
|---|---|---|
| Ingest | “get data from one system to another, with some modification to conform to the target system” | ► “You should reconsider whether you want a stream processing system or a simpler ingest-focused system like Kafka Connect. If you are sure, make sure it has both a good selection of connectors and high-quality connectors for the systems you are targeting.” |
| Low milliseconds actions | “Any application that requires almost immediate response. Some fraud-detection use cases.” | ► “You should also reconsider your choice of streams. Request-response patterns are often better suited. If you are sure, opt for one that supports an event-by-event low-latency model rather than one that focuses on microbatches.” |
| Asynchronous microservices | “perform a simple action on behalf of a larger business process, such as updating the inventory of a store… may need to maintain local state caching events as a way to improve performance.” | ► Need one that “integrates well with your message bus of choice (Kafka, hopefully), has change capture capabilities that easily deliver upstream changes to the microservice local state, and has good support of a local store that can serve as a cache or materialized view.” |
| Near real-time data analytics | “perform complex aggregations and joins in order to slice and dice the data and generate interesting, business-relevant insights.” | ► Need “great support for a local store — this time, not for local caches and materialized views but rather to support advanced aggregations, windows, and joins. The APIs should include support for custom aggregations, window operations, and multiple join types.” |
Note that two of the four answers are “you may not want stream processing at all.”
Note that two of the four answers are "you may not want stream processing at all." That's an unusually honest framing for a chapter selling a stream processing library.
7.2 Four global considerations
| Criterion | The questions to ask |
|---|---|
| Operability of the system | "Is it easy to deploy to production? Is it easy to monitor and troubleshoot? Is it easy to scale up and down? Does it integrate well with your existing infrastructure? What if there is a mistake and you need to REPROCESS data?" |
| Usability of APIs and ease of debugging | "I've seen ORDERS-OF-MAGNITUDE DIFFERENCES in the time it takes to write a high-quality application AMONG DIFFERENT VERSIONS OF THE SAME FRAMEWORK. Development time and time-to-market are important, so you need to choose a system that makes you efficient." |
| Makes hard things easy | "ALMOST EVERY SYSTEM WILL CLAIM they can do advanced windowed aggregations and maintain local stores, BUT THE QUESTION IS: DO THEY MAKE IT EASY FOR YOU? Do they handle GRITTY DETAILS around SCALE AND RECOVERY, OR DO THEY SUPPLY LEAKY ABSTRACTIONS AND MAKE YOU HANDLE MOST OF THE MESS?" |
| Community | "there's no replacement for a vibrant and active community. Good community means you get new features regularly, the quality is relatively good (no one wants to work on bad software), bugs get fixed quickly, and user questions get answers. It also means that if you get a strange error and Google it, YOU WILL FIND INFORMATION ABOUT IT because other people are using this system and seeing the same issues." |