Deterministic Simulation Testing for Consensus State Machines in Rust
Building a single-threaded seed-driven simulator for Raft + WAL storage: reproducing a 1-in-400,000 crash-recovery corruption bug in 11 seconds.
1. Why Real-Network Jepsen Tests Are Necessary But Too Slow
End-to-end fault injection on real Linux containers takes minutes to execute a few dozen partition and power-loss cycles. Inspired by FoundationDB and TigerBeetle, we abstracted our Rust storage engine behind three deterministic traits: LogicalClock, SimNetwork, and FaultyBlockDevice.
pub trait BlockEngine: Send + 'static {
fn submit_write(&mut self, offset: u64, data: &[u8], fua: bool) -> IoToken;
fn submit_fsync(&mut self) -> IoToken;
fn simulate_power_loss(&mut self, rng: &mut ChaCha8Rng);
}
2. Catching Torn Unsynced Directory Entries
Our simulator models an NVMe drive that reorders un-flushed sector writes and drops a random subset of dirty cache blocks on simulated power loss. Within 11 seconds (seed 0x94A11F0C382B0019), the simulator uncovered a bug where a newly rotated WAL segment file was fdatasync'd, but its parent directory fsync was deferred—causing the file to vanish after power loss even though the leader had already acknowledged the commit index to a peer.