How a fix in Go 1.9 sped up our Gitaly service by 30x

What this post added

This post details the debugging and resolution of performance degradation in the Gitaly service. The issue was identified as increasing latency over time, eventually leading to lockups. Through profiling (pprof) and cAdvisor monitoring, contention on `syscall.ForkLock` was pinpointed. Research led to the discovery that Go 1.9's switch from `fork/exec` to `posix_spawn` resolved this by reducing the overhead of process creation, especially with large virtual memory footprints. Recompiling Gitaly with Go 1.9 resulted in a 30x reduction in latency.