Post-Mortem: How a Single Slow Follower Stalled a Quorum Pipeline via TCP Receive Window Zeroing
When a degraded disk on 1 of 3 Raft nodes shouldn't affect quorum commits—but shared gRPC stream head-of-line blocking coupled the leader to the slowest peer.
1. Why a 2-of-3 Quorum Still Stalled on the 3rd Node
In a 3-node Raft group, committing a log entry only requires the leader + 1 healthy follower. Yet when follower `node-3` experienced an NVMe controller stall that froze its WAL write thread for 800 ms, the leader's commit latency for the entire cluster spiked to 420 ms.
2. Tracing the Coupling: Shared Serialization Mutex & TCP ZeroWindow
Packet captures showed `node-3` advertising `TCP ZeroWindow` (`win=0`) after its userspace gRPC receive buffer filled. On the leader, the replication broadcaster held a per-group batch lock while calling blocking stream `Send()` to each peer sequentially. Once `node-3`'s socket send buffer (`SO_SNDBUF`) filled, the broadcaster blocked inside the leader's replication loop before it could dispatch the next batch to the healthy follower `node-2`.
// Decoupled per-peer replication sender with non-blocking ring drop-to-snapshot fallback
select {
case peer.sendCh <- batch:
default:
peer.markLagging(batch.LastIndex) // Never block the leader's broadcast loop
}
Decoupling each follower into an independent non-blocking ring buffer that transitions lagging peers to asynchronous catch-up reading from the leader's WAL cache isolated slow-follower disk stalls completely.