Eliminating False Sharing in Lock-Free MPMC Queues on Zen 4 Multi-CCD Topologies
Using perf c2c HITM metrics to track down a 3.8x throughput regression in a Vyukov bounded MPMC queue across AMD EPYC Core Complex Dies (CCDs).
1. Why 64-Byte Alignment Was Not Enough
On a 32-core AMD EPYC 7543P / 9354P processor partitioned across 4 CCDs connected via the Infinity Fabric (IOD), our bounded lock-free MPMC ring queue scaled linearly within a single 8-core CCX (48M ops/sec) but collapsed to 14M ops/sec as soon as producers and consumers spanned two CCDs.
2. Hardware Prefetcher Adjacent Cache-Line Pulling
Although head_ and tail_ atomic indices were padded with alignas(64), the L2 stream prefetcher was speculatively pulling the adjacent 64-byte line (128-byte spatial pair), AND adjacent ring slots (each 32 bytes, containing sequence counter + payload pointer) shared a single 64-byte cache line between slot i and slot i+1.
template <typename T>
struct alignas(128) PaddedCell {
std::atomic<uint64_t> sequence;
T payload;
uint8_t pad[128 - sizeof(std::atomic<uint64_t>) - sizeof(T)];
};
struct alignas(128) QueueIndex {
std::atomic<uint64_t> value;
uint64_t local_cached_bound;
};
3. Index Shuffling (Cache-Line Permutation)
Rather than wasting 128 bytes per ring cell, we retained 32-byte cells but applied a low-bit XOR permutation to map logical index i to physical slot (i & ~7ULL) | ((i & 7ULL) * 2 % 8), ensuring consecutive sequence numbers land on distinct cache lines.
| Queue Layout (8P / 8C across 2 CCDs) | Remote HITM / 1M ops | Throughput | p99 Enqueue Latency |
|---|---|---|---|
| 64B padded head/tail, contiguous 32B cells | 184,200 | 14.1M ops/s | 1.42 µs |
| 128B padded head/tail + local bound cache | 92,400 | 28.7M ops/s | 0.64 µs |
| 128B head/tail + cache-line slot permutation | 11,300 | 53.4M ops/s | 0.19 µs |