Anatomy of a Go Runtime Scheduler Latency Spike: Netpoller, Syscall P Hand-offs, and GOMAXPROCS

Using go tool trace and runtime/metrics (/sched/latencies:seconds) to diagnose a 15 ms tail latency anomaly in a containerized Go proxy.

1. Distinguishing GC Stalls From Runqueue Scheduling Delay

When our Go control-plane proxy exhibited 15 ms p999 response spikes, initial suspicion fell on garbage collection. However, exporting /sched/latencies:seconds and /gc/pauses:seconds from Go's runtime/metrics package showed GC STW pauses never exceeded 190 µs—whereas runnable goroutines occasionally waited 14.8 ms in a local P runqueue before executing.

2. Container CFS Quota Mismatch + Blocking cgo/Syscall Hand-offs

The container was deployed on a 64-core host with a Kubernetes CPU limit of 4000m (4 cores), without setting GOMAXPROCS (or before container-aware defaults were active). With GOMAXPROCS=64, 64 OS threads woke up simultaneously on bursts, consumed the entire 400 ms CFS quota inside the first 6.25 ms of the 100 ms period, and got throttled by the Linux scheduler for the remaining 93.75 ms. Matching GOMAXPROCS=4 and replacing synchronous flock syscalls collapsed p999 scheduling delay to 0.4 ms.

← Older Note (2025-05-15)Bloom Filter False Positives vs. Ribbon Filters in Memory-Constrained SSTable CachesNewer Note (2025-08-11) →Understanding ext4 JBD2 Journal Commit Bottlenecks Under Concurrent fdatasync

← Back to all notes · View full chronological archive (24)