Measuring PCIe Peer-to-Peer DMA Throughput Between NVMe Drives and RDMA NICs

Evaluating Linux pci_p2pdma and CMB (Controller Memory Buffer) transfers across PCIe root complexes vs. dedicated PCIe switches.

1. Topology Matters: Root Complex Hairpin vs. PCIe Switch

When serving bulk SSTable replication snapshots over RoCEv2 (25GbE/100GbE), standard transfers DMA data from the NVMe SSD into host DRAM over the CPU Root Complex, and then DMA the same pages back out from DRAM to the RDMA NIC—consuming 2x memory bus bandwidth.

PCIe Path TopologySustained TransferHost DRAM BW Usedp99 Chunk Latency
Standard Host DRAM bounce buffer6.4 GB/s12.8 GB/s410 µs
P2P DMA across CPU Root Complex3.9 GB/s (throttled)0.4 GB/s690 µs
P2P DMA under same PCIe 4.0 Switch6.8 GB/s0.2 GB/s320 µs

On AMD EPYC platforms, peer-to-peer DMA must traverse devices behind the same physical PCIe switch or root port (`pci_p2pdma_distance` ≤ 2); hairpinning P2P TLPs through the CPU root complex incurs severe credit throttling.

← Older Note (2024-11-24)B-Epsilon Trees vs. Write-Optimized B+Trees for High-Ingestion Secondary IndexesNewer Note (2025-02-18) →Wait-Free Epoch-Based Reclamation Without Tail Latency Spikes From Stalled Readers

← Back to all notes · View full chronological archive (24)