Profiling Hardware TLB Misses with perf stat and 2MiB Hugetlbfs Backing Pools
Measuring page-table walk cycles on a 48 GiB in-memory hash table using perf dTLB-load-misses and MAP_HUGETLB.
1. Counting Page-Walk Cycles on x86_64
At 4 KiB page granularity, covering a 48 GiB open-addressed hash table requires 12,582,912 page table entries—far exceeding the ~1,536-entry L2 dTLB capacity of modern cores. Every random hash probe triggers a 4-level radix page table walk.
$ perf stat -e dTLB-loads,dTLB-load-misses,dtlb_load_misses.walk_completed,cycles -p $(pgrep kv-engine) -- sleep 10
4,812,490,112 dTLB-loads
612,840,229 dTLB-load-misses # 12.73% of all dTLB cache accesses
589,110,402 dtlb_load_misses.walk_completed
Backing the primary hash bucket array with explicit 2 MiB huge pages (`mmap` with `MAP_HUGETLB | (21 << MAP_HUGE_SHIFT)`) reduced the required TLB entries to 24,576, dropping page-walk cycles from 16.4% to 0.9% and boosting lookup throughput by 29%.