Tracing Ephemeral Port Exhaustion and TIME_WAIT Reuse Anomalies Behind L4 Load Balancers
Why connect(2) returned EADDRNOTAVAIL despite tcp_tw_reuse=1, and how IP_BIND_ADDRESS_NO_PORT + multi-IP source binding scaled outbound pools to 500k conns.
1. Why tcp_tw_reuse Fails When Destination Timestamps Reset
Linux's net.ipv4.tcp_tw_reuse=1 allows an outbound connect(2) to reclaim a 4-tuple in TIME_WAIT only if the new TCP timestamp is strictly greater than the latest timestamp recorded on the old connection (and at least 1 second has elapsed). Worse, inside __inet_hash_connect(), calling bind(fd, src_ip, 0) before connect() forces the kernel to pick a source port before knowing the destination (dst_ip, dst_port), preventing port sharing across distinct backend servers!
int one = 1;
setsockopt(fd, SOL_IP, IP_BIND_ADDRESS_NO_PORT, &one, sizeof(one));
bind(fd, (struct sockaddr *)&src_addr, sizeof(src_addr)); // Port deferred to connect()
Enabling IP_BIND_ADDRESS_NO_PORT lets connect(2) select the ephemeral port with full 4-tuple knowledge, multiplying usable outbound concurrency by the number of distinct backend endpoints.