14.6 Stream processing use cases
"stream processing — or continuous processing — is useful in cases where we want our events to be processed in quick order rather than wait for hours until the next batch, but also where we are NOT EXPECTING A RESPONSE TO ARRIVE IN MILLISECONDS."
Customer service — the hotel story
"Suppose we just reserved a room at a large hotel chain... A few minutes later, when the confirmation still hasn't arrived, we call customer service. Suppose the desk tells us: 'I don't see the order in our system, but THE BATCH JOB that loads the data from the reservation system to the hotels and the customer service desk ONLY RUNS ONCE A DAY, so please call back tomorrow. You should see the email within 2–3 business days.' This doesn't sound like very good service, yet WE'VE HAD THIS CONVERSATION MORE THAN ONCE WITH A LARGE HOTEL CHAIN."
What you actually want: "EVERY SYSTEM in the hotel chain to get an update about a new reservation SECONDS OR MINUTES after the reservation is made — including the customer service center, the hotel, the email system, the website... the customer service center to be able to immediately pull up all the details about ANY of our PAST VISITS to ANY of the hotels, and the reception desk to know that we are a loyal customer so they can give us an upgrade."
Internet of Things — predictive maintenance
*"A very common use case... is to try to PREDICT WHEN PREVENTIVE MAINTENANCE IS NEEDED. This is similar to application monitoring but applied to HARDWARE, and is common in manufacturing, telecommunications (identifying faulty cellphone towers), cable TV (identifying faulty box-top devices BEFORE USERS COMPLAIN)...
The goal: process events arriving from devices at a large scale and IDENTIFY PATTERNS THAT SIGNAL THAT A DEVICE REQUIRES MAINTENANCE. These patterns can be dropped packets for a switch, MORE FORCE REQUIRED TO TIGHTEN SCREWS in manufacturing, or USERS RESTARTING THE BOX MORE FREQUENTLY for cable TV."*
Fraud detection / anomaly detection
"detecting credit card fraud, stock trading fraud, video-game cheaters, and cybersecurity risks. In all these fields, there are large benefits to catching fraud as early as possible, so a near real-time system that is capable of responding quickly — PERHAPS STOPPING A BAD TRANSACTION BEFORE IT IS EVEN APPROVED — is much preferred to a batch job that detects fraud THREE DAYS AFTER THE FACT, when cleanup is much more complicated."
💡 The beaconing example — a genuinely instructive one:
*"In cybersecurity, there is a method known as BEACONING. When the hacker plants malware inside the organization, it will OCCASIONALLY REACH OUTSIDE TO RECEIVE COMMANDS. IT CAN BE DIFFICULT TO DETECT THIS ACTIVITY SINCE IT CAN HAPPEN AT ANY TIME AND ANY FREQUENCY.
Typically, NETWORKS ARE WELL DEFENDED AGAINST EXTERNAL ATTACKS BUT MORE VULNERABLE TO SOMEONE INSIDE THE ORGANIZATION REACHING OUT.
By processing the large stream of network connection events and recognizing a PATTERN of communication as ABNORMAL (for example, detecting that THIS HOST TYPICALLY DOESN'T ACCESS THOSE SPECIFIC IPs), the security organization can be alerted early, before more harm is done."*