Debugging P999 Write Stalls in LSM-Tree Storage Engines Under Skewed Zipfian Workloads

Tracing a recurring 180 ms latency spike every 14 minutes in a 4 TB RocksDB/Pebble cluster: how L0-to-L1 compaction debt, memtable flush backpressure, and NVMe GC interacted.

1. The 14-Minute Latency Sawtooth

Under a 140,000 upsert/sec Zipfian workload on a 4 TB LSM dataset, median write latency stayed at 310 µs, but every 14 minutes p999 jumped from 6.5 ms to 184 ms. Correlating iostat -xz 1 with compaction logs showed 64 MiB SSTable fdatasync bursts colliding with NVMe erase-block reclamation.

Device       r/s     rkB/s   r_await     w/s     wkB/s   w_await  aqu-sz  %util
nvme0n1   1842.0  117888.0      0.42  6410.0  582400.0     19.84  127.94  99.60

2. Token-Bucket Compaction Pacing & Incremental sync_file_range

ConfigurationWAFWAL Sync p99Client Write p999
Unthrottled 8 compaction threads11.414.2 ms184.0 ms
+ 2 MiB incremental sync_file_range11.15.8 ms48.5 ms
+ 320 MB/s token-bucket compaction rate limit4.20.94 ms6.8 ms
← Older Note (2026-07-02)Why Our eBPF Ring Buffer Dropped 4% of Socket Events at 800k eps (And How We Fixed It)Newer Note (2026-09-27) →FQ Pacing Granularity and BBR Bandwidth Probing on Shallow-Buffer WAN Links

← Back to all notes · View full chronological archive (24)