Why Our eBPF Ring Buffer Dropped 4% of Socket Events at 800k eps (And How We Fixed It)
Investigating silent ring buffer reservation failures in high-throughput telemetry: profiling cache-line bouncing on producer_pos and tuning adaptive BPF_RB_NO_WAKEUP watermarks.
1. Reserve Failures With 70% Free Ring Space
When our 32-core telemetry agent exceeded 800,000 events/sec, bpf_ringbuf_reserve() failed on 4.1% of events despite a 64 MiB ring buffer. Running perf c2c revealed severe cross-socket cache-line bouncing on the shared producer_pos spinlock and excessive epoll wakeup IPIs.
long pending = bpf_ringbuf_query(&rb, BPF_RB_AVAIL_DATA);
long flags = (pending >= 65536) ? BPF_RB_FORCE_WAKEUP : BPF_RB_NO_WAKEUP;
bpf_ringbuf_submit(e, flags);
Batching wakeups to a 64 KiB watermark and sharding the ring buffer per NUMA node via bpf_get_numa_node_id() eliminated all drops up to 1.45M events/sec.