9. The Trouble with Distributed Systems
9.9 Production failure catalog for this chapter
| Symptom | Underlying mechanism |
|---|---|
| Request timed out; did it happen or not? | Six indistinguishable cases (§2.1) — you cannot know |
| Retry caused a duplicate charge/email | Timeout ambiguity + non-idempotent operation |
| TCP said "delivered," the work never happened | ACK means the KERNEL received it, not the application |
| Data duplicated after a reconnect | TCP dedup applies to one connection only |
| Redundant switches, still an outage | Redundancy doesn't guard against human misconfiguration |
| A can reach B, B can reach C, A can't reach C | Partial/asymmetric network fault |
| Node sends fine but receives nothing (or vice versa) | One-directional link failure |
| Cluster deadlocked and stayed broken after the network recovered | Untested network-fault error handling |
| Healthy node evicted during a traffic spike | Timeout too short; detecting overload as death |
| All nodes declared each other dead | Cascading failure from premature eviction |
| Latency fine at 60% load, chaotic at 90% | Queueing delays explode near capacity |
| Random slowness with no code change | Noisy neighbour in a multitenant environment |
| Timestamps out of order across nodes | Clock skew; ordering by wall clock |
| Writes silently disappear, no errors | LWW + a node with a lagging clock |
| Elapsed time computed as negative | Time-of-day clock stepped backward |
| Everything hung at midnight | Leap second |
| Clock drifted for weeks, nobody noticed | NTP firewalled off — the silent failure |
| A microsecond timestamp that's wrong by 40 ms | No confidence interval exposed |
| Leader kept writing after losing its lease | Process pause → zombie; no fencing |
| Ancient write arrives and corrupts a file | Delayed packet from a crashed former leaseholder |
| STONITH fired and both nodes died | Mutual shutdown; STONITH doesn't handle delayed packets |
| Node responds to health checks but does no work | Fail-slow / gray failure / limping node |
| Quorum algorithm violated after a disk wipe | Node amnesia breaks the stable-storage assumption |
| Verified model, buggy system | Spec/implementation drift |
| Simulation replay isn't reproducible | An uncontrolled nondeterminism source (hash order, OOM) |