Transparent Huge Pages, jemalloc Arena Fragmentation, and MADV_COLLAPSE in Long-Lived Daemons
Balancing dTLB hit rates against RSS bloat in multi-day storage nodes using jemalloc metadata_thp and selective MADV_COLLAPSE.
1. The Classic THP Dilemma: Global Always vs. madvise
Setting system-wide /sys/kernel/mm/transparent_hugepage/enabled to always often inflates RSS by 25–40% in long-lived key-value engines due to internal fragmentation inside 2 MiB pages when sparse small allocations hold entire huge pages resident. Conversely, disabling THP completely increased dTLB load miss cycles on our 64 GiB block cache to 14.8% of total CPU time.
2. Tuning jemalloc dirty_decay_ms and MADV_COLLAPSE
export MALLOC_CONF="background_thread:true,metadata_thp:auto,dirty_decay_ms:5000,muzzy_decay_ms:10000,narenas:16"
By leaving system THP in madvise mode, enabling metadata_thp:auto for jemalloc's internal page-map trees, and explicitly invoking madvise(arena_ptr, len, MADV_COLLAPSE) on our pre-allocated 32 GiB block cache and `.text` executable segments at startup, we cut dTLB walk stalls by 62% while keeping allocator fragmentation under 3.5% over a 30-day run.