9. The Trouble with Distributed Systems

"They're funny things, Accidents. You never have them till you're having them." — A.A. Milne

The mindset shift this chapter demands:

If you want your system to be reliable in the presence of faults, you have to RADICALLY CHANGE YOUR MINDSET and focus on what could go wrong, even though it may be unlikely. It doesn't matter whether there is only a one-in-a-million chance; IN A LARGE ENOUGH SYSTEM, ONE-IN-A-MILLION EVENTS HAPPEN EVERY DAY. Experienced systems operators will tell you that ANYTHING THAT CAN GO WRONG WILL GO WRONG.

In distributed systems, SUSPICION, PESSIMISM, AND PARANOIA PAY OFF.

The chapter is deliberately all problems. Ch 10 is the solutions.

1. Unreliable networks — packets are lost and delayed arbitrarily, and you cannot tell why you got no response.
2. Unreliable clocks — drift, jumps, and no confidence interval.
3. Process pauses — a thread can be frozen for minutes, mid-function, without noticing.
4. Knowledge and truth — quorums, leases, fencing, Byzantine faults.
5. System models — formalizing what we assume; safety versus liveness; and how we test any of this.

---