XDP and AF_XDP Zero-Copy Packet Steering Across Multi-Queue ConnectX-6 NICs
Using hardware RSS flow steering rules and XDP_REDIRECT to route custom UDP telemetry traffic directly into userspace UMEM rings at 19.4 Mpps per core.
1. Bypassing the sk_buff Allocation Path
At 14.88 Mpps (10GbE minimum-sized frame line rate), allocating and freeing kernel sk_buff structures consumes more than 65% of a single core's cycle budget. AF_XDP in zero-copy (XDP_ZEROCOPY) mode programs the NIC's DMA engine to write incoming frames directly into page-pinned userspace UMEM chunks.
SEC("xdp")
int xdp_telemetry_redirect(struct xdp_md *ctx) {
void *data = (void *)(long)ctx->data;
void *data_end = (void *)(long)ctx->data_end;
if (is_telemetry_udp_port(data, data_end, 9104)) {
int rx_queue = ctx->rx_queue_index;
return bpf_redirect_map(&xsks_map, rx_queue, XDP_PASS);
}
return XDP_PASS; // Normal SSH/TCP traffic continues to Linux network stack
}
Pairing the XDP program with ethtool -N eth0 flow-type udp4 dst-port 9104 action 8 isolates telemetry packets onto dedicated hardware RX queues bound to poll-mode AF_XDP sockets, leaving regular kernel TCP sockets completely undisturbed.