Debugging P999 Write Stalls in LSM-Tree Storage Engines Under Skewed Zipfian Workloads
Tracing a recurring 180 ms latency spike every 14 minutes in a 4 TB RocksDB/Pebble cluster: how L0-to-L1 compaction debt, memtable flush backpressure, and NVMe GC interacted.
1. The 14-Minute Latency Sawtooth
Under a 140,000 upsert/sec Zipfian workload on a 4 TB LSM dataset, median write latency stayed at 310 µs, but every 14 minutes p999 jumped from 6.5 ms to 184 ms. Correlating iostat -xz 1 with compaction logs showed 64 MiB SSTable fdatasync bursts colliding with NVMe erase-block reclamation.
Device r/s rkB/s r_await w/s wkB/s w_await aqu-sz %util
nvme0n1 1842.0 117888.0 0.42 6410.0 582400.0 19.84 127.94 99.60
2. Token-Bucket Compaction Pacing & Incremental sync_file_range
| Configuration | WAF | WAL Sync p99 | Client Write p999 |
|---|---|---|---|
| Unthrottled 8 compaction threads | 11.4 | 14.2 ms | 184.0 ms |
| + 2 MiB incremental sync_file_range | 11.1 | 5.8 ms | 48.5 ms |
| + 320 MB/s token-bucket compaction rate limit | 4.2 | 0.94 ms | 6.8 ms |