Profiling Hardware TLB Misses with perf stat and 2MiB Hugetlbfs Backing Pools

Measuring page-table walk cycles on a 48 GiB in-memory hash table using perf dTLB-load-misses and MAP_HUGETLB.

1. Counting Page-Walk Cycles on x86_64

At 4 KiB page granularity, covering a 48 GiB open-addressed hash table requires 12,582,912 page table entries—far exceeding the ~1,536-entry L2 dTLB capacity of modern cores. Every random hash probe triggers a 4-level radix page table walk.

$ perf stat -e dTLB-loads,dTLB-load-misses,dtlb_load_misses.walk_completed,cycles -p $(pgrep kv-engine) -- sleep 10
       4,812,490,112      dTLB-loads
         612,840,229      dTLB-load-misses          #   12.73% of all dTLB cache accesses
         589,110,402      dtlb_load_misses.walk_completed

Backing the primary hash bucket array with explicit 2 MiB huge pages (`mmap` with `MAP_HUGETLB | (21 << MAP_HUGE_SHIFT)`) reduced the required TLB entries to 24,576, dropping page-walk cycles from 16.4% to 0.9% and boosting lookup throughput by 29%.

← Older Note (2024-08-30)Adaptive Concurrency Limits Using Little's Law and Gradient2 Queue ControllersNewer Note (2024-11-24) →B-Epsilon Trees vs. Write-Optimized B+Trees for High-Ingestion Secondary Indexes

← Back to all notes · View full chronological archive (24)