Dissecting Linux Page Cache Readahead Thrashing Under Random mmap Lookups

How default 128 KiB kernel readahead amplified NVMe read bandwidth by 16x on a memory-mapped B-tree index, and why MADV_RANDOM + explicit MADV_WILLNEED fixed it.

1. When 4 KiB Point Lookups Pull 128 KiB from Flash

When an application maps a 500 GiB index file with mmap(2) and performs uniform random 256-byte point lookups without calling madvise(addr, len, MADV_RANDOM), the Linux VMA fault handler (filemap_fault → do_sync_mmap_readahead) speculatively schedules up to 32 pages (128 KiB) of readahead per major page fault.

// Disable speculative sequential readahead on the mapped index region
if (madvise(mmap_base, file_size, MADV_RANDOM) != 0) {
    perror("madvise MADV_RANDOM");
}
// For range scans, explicitly prefetch the target leaf extent:
madvise(leaf_aligned_ptr, extent_len, MADV_WILLNEED);

Applying MADV_RANDOM globally and issuing targeted MADV_WILLNEED hints only before multi-key range cursors dropped physical NVMe read bandwidth from 2.4 GB/s (saturated) to 155 MB/s while tripling query throughput.

← Older Note (2025-11-03)XDP and AF_XDP Zero-Copy Packet Steering Across Multi-Queue ConnectX-6 NICsNewer Note (2026-01-29) →Deterministic Simulation Testing for Consensus State Machines in Rust

← Back to all notes · View full chronological archive (24)