XDP and AF_XDP Zero-Copy Packet Steering Across Multi-Queue ConnectX-6 NICs

Using hardware RSS flow steering rules and XDP_REDIRECT to route custom UDP telemetry traffic directly into userspace UMEM rings at 19.4 Mpps per core.

1. Bypassing the sk_buff Allocation Path

At 14.88 Mpps (10GbE minimum-sized frame line rate), allocating and freeing kernel sk_buff structures consumes more than 65% of a single core's cycle budget. AF_XDP in zero-copy (XDP_ZEROCOPY) mode programs the NIC's DMA engine to write incoming frames directly into page-pinned userspace UMEM chunks.

SEC("xdp")
int xdp_telemetry_redirect(struct xdp_md *ctx) {
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;
    if (is_telemetry_udp_port(data, data_end, 9104)) {
        int rx_queue = ctx->rx_queue_index;
        return bpf_redirect_map(&xsks_map, rx_queue, XDP_PASS);
    }
    return XDP_PASS; // Normal SSH/TCP traffic continues to Linux network stack
}

Pairing the XDP program with ethtool -N eth0 flow-type udp4 dst-port 9104 action 8 isolates telemetry packets onto dedicated hardware RX queues bound to poll-mode AF_XDP sockets, leaving regular kernel TCP sockets completely undisturbed.

← Older Note (2025-09-22)SIMD-Accelerated Varint and Delta Decoding with AVX-512 in Columnar BlocksNewer Note (2025-12-14) →Dissecting Linux Page Cache Readahead Thrashing Under Random mmap Lookups

← Back to all notes · View full chronological archive (24)