Measuring PCIe Peer-to-Peer DMA Throughput Between NVMe Drives and RDMA NICs
Evaluating Linux pci_p2pdma and CMB (Controller Memory Buffer) transfers across PCIe root complexes vs. dedicated PCIe switches.
1. Topology Matters: Root Complex Hairpin vs. PCIe Switch
When serving bulk SSTable replication snapshots over RoCEv2 (25GbE/100GbE), standard transfers DMA data from the NVMe SSD into host DRAM over the CPU Root Complex, and then DMA the same pages back out from DRAM to the RDMA NIC—consuming 2x memory bus bandwidth.
| PCIe Path Topology | Sustained Transfer | Host DRAM BW Used | p99 Chunk Latency |
|---|---|---|---|
| Standard Host DRAM bounce buffer | 6.4 GB/s | 12.8 GB/s | 410 µs |
| P2P DMA across CPU Root Complex | 3.9 GB/s (throttled) | 0.4 GB/s | 690 µs |
| P2P DMA under same PCIe 4.0 Switch | 6.8 GB/s | 0.2 GB/s | 320 µs |
On AMD EPYC platforms, peer-to-peer DMA must traverse devices behind the same physical PCIe switch or root port (`pci_p2pdma_distance` ≤ 2); hairpinning P2P TLPs through the CPU root complex incurs severe credit throttling.