Measuring NVMe Garbage Collection Tail Latency and Append-Only WAL Alignment
Why pre-allocating WAL segments and aligning O_DIRECT appends to 64 KiB Indirection Units (IU) cuts p999 fsync latency by 94% on datacenter NVMe drives.
1. FTL Indirection Units and Partial-Block Read-Modify-Write
High-capacity datacenter TLC NVMe SSDs use a 16 KiB or 64 KiB Indirection Unit (IU). Growing a WAL file via buffered 4 KiB writes forces both extent tree metadata updates and FTL garbage collection stalls.
| WAL Pipeline (4 KiB batch, 80% SSD fill) | p50 | p99 | p999 | Max Spike |
|---|---|---|---|---|
| Buffered write() + fsync() (growing file) | 145 µs | 1.92 ms | 14.60 ms | 41.2 ms |
| O_DIRECT pwrite() + fdatasync() (fallocated) | 82 µs | 0.48 ms | 2.15 ms | 8.4 ms |
| Recycled segment + WRITE_FIXED + FSYNC_DATASYNC | 64 µs | 0.29 ms | 0.85 ms | 2.3 ms |