Deterministic Simulation Testing for Consensus State Machines in Rust

Building a single-threaded seed-driven simulator for Raft + WAL storage: reproducing a 1-in-400,000 crash-recovery corruption bug in 11 seconds.

1. Why Real-Network Jepsen Tests Are Necessary But Too Slow

End-to-end fault injection on real Linux containers takes minutes to execute a few dozen partition and power-loss cycles. Inspired by FoundationDB and TigerBeetle, we abstracted our Rust storage engine behind three deterministic traits: LogicalClock, SimNetwork, and FaultyBlockDevice.

pub trait BlockEngine: Send + 'static {
    fn submit_write(&mut self, offset: u64, data: &[u8], fua: bool) -> IoToken;
    fn submit_fsync(&mut self) -> IoToken;
    fn simulate_power_loss(&mut self, rng: &mut ChaCha8Rng);
}

2. Catching Torn Unsynced Directory Entries

Our simulator models an NVMe drive that reorders un-flushed sector writes and drops a random subset of dirty cache blocks on simulated power loss. Within 11 seconds (seed 0x94A11F0C382B0019), the simulator uncovered a bug where a newly rotated WAL segment file was fdatasync'd, but its parent directory fsync was deferred—causing the file to vanish after power loss even though the leader had already acknowledged the commit index to a peer.

← Older Note (2025-12-14)Dissecting Linux Page Cache Readahead Thrashing Under Random mmap LookupsNewer Note (2026-03-08) →Transparent Huge Pages, jemalloc Arena Fragmentation, and MADV_COLLAPSE in Long-Lived Daemons

← Back to all notes · View full chronological archive (24)