Dissecting Linux Page Cache Readahead Thrashing Under Random mmap Lookups
How default 128 KiB kernel readahead amplified NVMe read bandwidth by 16x on a memory-mapped B-tree index, and why MADV_RANDOM + explicit MADV_WILLNEED fixed it.
1. When 4 KiB Point Lookups Pull 128 KiB from Flash
When an application maps a 500 GiB index file with mmap(2) and performs uniform random 256-byte point lookups without calling madvise(addr, len, MADV_RANDOM), the Linux VMA fault handler (filemap_fault → do_sync_mmap_readahead) speculatively schedules up to 32 pages (128 KiB) of readahead per major page fault.
// Disable speculative sequential readahead on the mapped index region
if (madvise(mmap_base, file_size, MADV_RANDOM) != 0) {
perror("madvise MADV_RANDOM");
}
// For range scans, explicitly prefetch the target leaf extent:
madvise(leaf_aligned_ptr, extent_len, MADV_WILLNEED);
Applying MADV_RANDOM globally and issuing targeted MADV_WILLNEED hints only before multi-key range cursors dropped physical NVMe read bandwidth from 2.4 GB/s (saturated) to 155 MB/s while tripling query throughput.