Measuring NVMe Garbage Collection Tail Latency and Append-Only WAL Alignment

Why pre-allocating WAL segments and aligning O_DIRECT appends to 64 KiB Indirection Units (IU) cuts p999 fsync latency by 94% on datacenter NVMe drives.

1. FTL Indirection Units and Partial-Block Read-Modify-Write

High-capacity datacenter TLC NVMe SSDs use a 16 KiB or 64 KiB Indirection Unit (IU). Growing a WAL file via buffered 4 KiB writes forces both extent tree metadata updates and FTL garbage collection stalls.

WAL Pipeline (4 KiB batch, 80% SSD fill)p50p99p999Max Spike
Buffered write() + fsync() (growing file)145 µs1.92 ms14.60 ms41.2 ms
O_DIRECT pwrite() + fdatasync() (fallocated)82 µs0.48 ms2.15 ms8.4 ms
Recycled segment + WRITE_FIXED + FSYNC_DATASYNC64 µs0.29 ms0.85 ms2.3 ms
← Older Note (2026-04-12)Eliminating False Sharing in Lock-Free MPMC Queues on Zen 4 Multi-CCD TopologiesNewer Note (2026-07-02) →Why Our eBPF Ring Buffer Dropped 4% of Socket Events at 800k eps (And How We Fixed It)

← Back to all notes · View full chronological archive (24)