How we reduced 502 errors by caring about PID 1 in Kubernetes

This post details the debugging and resolution of intermittent 502 errors impacting the GitLab Pages service, specifically caused by GitLab Workhorse failing to shut down gracefully during Kubernetes pod termination. The investigation revealed that Workhorse was not correctly handling the SIGTERM signal, leading to Kubernetes having to forcefully kill the process after the default 30-second grace period. This prolonged the period where the pod was still in the Kubernetes service endpoint but no longer functional, resulting in 502 errors. The fix involved ensuring Workhorse correctly handles SIGTERM and exits within the grace period, preventing the readiness probe failures from causing extended service disruption.