Anatomy of a Go Runtime Scheduler Latency Spike: Netpoller, Syscall P Hand-offs, and GOMAXPROCS
Using go tool trace and runtime/metrics (/sched/latencies:seconds) to diagnose a 15 ms tail latency anomaly in a containerized Go proxy.
1. Distinguishing GC Stalls From Runqueue Scheduling Delay
When our Go control-plane proxy exhibited 15 ms p999 response spikes, initial suspicion fell on garbage collection. However, exporting /sched/latencies:seconds and /gc/pauses:seconds from Go's runtime/metrics package showed GC STW pauses never exceeded 190 µs—whereas runnable goroutines occasionally waited 14.8 ms in a local P runqueue before executing.
2. Container CFS Quota Mismatch + Blocking cgo/Syscall Hand-offs
The container was deployed on a 64-core host with a Kubernetes CPU limit of 4000m (4 cores), without setting GOMAXPROCS (or before container-aware defaults were active). With GOMAXPROCS=64, 64 OS threads woke up simultaneously on bursts, consumed the entire 400 ms CFS quota inside the first 6.25 ms of the 100 ms period, and got throttled by the Linux scheduler for the remaining 93.75 ms. Matching GOMAXPROCS=4 and replacing synchronous flock syscalls collapsed p999 scheduling delay to 0.4 ms.