13. A Philosophy of Streaming Systems
13.7 Production failure catalog for this chapter
| Symptom | Underlying mechanism |
|---|---|
| Database and search index permanently disagree | App writes to both — neither is "in charge" of ordering |
| Ex-partner receives the message they shouldn't | Causal dependency lost across two services with no shared order |
| Ordering ambiguous between two regions | Total order broadcast doesn't scale past one leader |
| One failing component takes down the whole system | Synchronous distributed transactions amplify local faults |
| Schema migration is a terrifying all-or-nothing cutover | No side-by-side derived views; irreversibility |
| Reprocessing produces different numbers than the original run | Nondeterministic derivation (time-dependent join, external call) |
| Ten pieces of infrastructure, one small team, constant incidents | Premature unbundling — "a form of premature optimization" |
| $22 transferred instead of $11 | Non-idempotent transaction + user retry past every dedup layer |
| Two accounts created with the same username | Uniqueness enforced without consensus, or async multi-leader |
| Multishard transfer half-applied | No request-ID dedup, or a nondeterministic processor |
| Money stuck permanently "reserved" | Lost/undelivered downstream event with no sweeper |
| Credits and debits don't sum to zero | Integrity violation — permanent, needs explicit repair |
| Derived store silently drifted from the source | No reconciliation / end-to-end integrity check |
| Backup found to be broken during a real incident | Never restore-tested |
| Corruption present in every retained backup | Silent corruption undetected for longer than the retention window |
| Uniqueness constraint violated by the database itself | A database bug — MySQL has done this |
| "Serializable" isolation exhibited write skew | A database bug — PostgreSQL has done this |
| App uses weak isolation unsafely; DB "consistency" meaningless | ACID consistency assumes bug-free transactions |
| Cannot explain why a set of rows changed | Mutation log without the intent; application logic was transient |
| UI shows stale data until reload | Read path only; write path never extended to the client |