6.9 Worked examples
This is why quorums are seldom more than 4-of-7 or 5-of-9.
① Quorum arithmetic. n=5. Which (w,r) satisfy w+r>n, and what does each tolerate?
| w | r | w+r | Valid? | Write availability | Read availability |
|---|---|---|---|---|---|
| 3 | 3 | 6 | ✔ | tolerates 2 down | tolerates 2 down |
| 5 | 1 | 6 | ✔ | 0 down | tolerates 4 down |
| 1 | 5 | 6 | ✔ | tolerates 4 down | 0 down |
| 4 | 2 | 6 | ✔ | tolerates 1 down | tolerates 3 down |
| 2 | 2 | 4 | ✗ | tolerates 3 down | tolerates 3 down — stale reads possible |
② Failover data loss. Async replication, leader committing 5,000 writes/s, follower lag p99 = 800 ms. Leader dies. Expected loss on promotion ≈ 5,000 × 0.8 = 4,000 writes. With semisync (one sync follower), loss on promotion of that follower = 0; on promotion of an async follower, unchanged. This calculation is the entire argument for semisync.
③ Read-after-write window. Replication lag p99 = 300 ms; a user's page reload happens ~200 ms after submit. Without mitigation, roughly the p99 fraction of users hitting a lagging replica see stale data — and since users reload immediately, this is not a rare event, it's the common path. Reading from the leader for 1 second after a write eliminates it at the cost of routing ~1 leader-read per write.
④ Tail latency vs quorum size. If each replica has an independent 1% chance of being slow (>100 ms), then waiting for the fastest r of n:
r=2, n=3: P(at least 2 of 3 fast) — slow response only if ≥2 replicas are slow ≈ 3×(0.01)² ≈ 0.03%r=5, n=9: needs 5 fast of 9; slow if ≥5 slow — negligible, but you now wait for the 5th-fastest rather than the 2nd-fastest, so median latency rises. This is why quorums are seldom more than 4-of-7 or 5-of-9.
⑤ Version-vector siblings. Three replicas, client reads at {A:2, B:1, C:1} and writes back. Meanwhile another client wrote on B, producing {A:2, B:2, C:1}. Comparing: neither vector dominates the other only if one has a strictly greater entry somewhere and strictly lesser elsewhere. Here {A:2,B:2,C:1} dominates {A:2,B:1,C:1} (≥ in all positions, > in one) ⇒ not concurrent; the B write happened after. Now add a third: {A:3,B:1,C:1} vs {A:2,B:2,C:1} — neither dominates ⇒ concurrent ⇒ siblings.
⑥ Hinted-handoff load. A node is offline for 4 hours in a 3-replica cluster taking 10,000 writes/s. Each write it missed is stored as a hint on another node: ~144 million hints, which must all be delivered when it returns — on top of normal traffic, to a node that is already cold. This is exactly the "additional load at a time when the system is already under strain" the book warns about, and it's why hint TTLs exist.