Tracing Ephemeral Port Exhaustion and TIME_WAIT Reuse Anomalies Behind L4 Load Balancers

Why connect(2) returned EADDRNOTAVAIL despite tcp_tw_reuse=1, and how IP_BIND_ADDRESS_NO_PORT + multi-IP source binding scaled outbound pools to 500k conns.

1. Why tcp_tw_reuse Fails When Destination Timestamps Reset

Linux's net.ipv4.tcp_tw_reuse=1 allows an outbound connect(2) to reclaim a 4-tuple in TIME_WAIT only if the new TCP timestamp is strictly greater than the latest timestamp recorded on the old connection (and at least 1 second has elapsed). Worse, inside __inet_hash_connect(), calling bind(fd, src_ip, 0) before connect() forces the kernel to pick a source port before knowing the destination (dst_ip, dst_port), preventing port sharing across distinct backend servers!

int one = 1;
setsockopt(fd, SOL_IP, IP_BIND_ADDRESS_NO_PORT, &one, sizeof(one));
bind(fd, (struct sockaddr *)&src_addr, sizeof(src_addr)); // Port deferred to connect()

Enabling IP_BIND_ADDRESS_NO_PORT lets connect(2) select the ephemeral port with full 4-tuple knowledge, multiplying usable outbound concurrency by the number of distinct backend endpoints.

← Older Note (2025-02-18)Wait-Free Epoch-Based Reclamation Without Tail Latency Spikes From Stalled ReadersNewer Note (2025-05-15) →Bloom Filter False Positives vs. Ribbon Filters in Memory-Constrained SSTable Caches

← Back to all notes · View full chronological archive (24)