6 Lessons we learned when debugging a scaling problem on GitLab.com

What this post added

This post details debugging a scaling problem on GitLab.com related to intermittent SSH connection errors during Git pulls. The investigation involved packet captures, log analysis (HAProxy and SSHD), and identifying a pattern of errors clustering around the top of the minute, attributed to scheduled jobs hitting the `MaxStartups` limit on SSHD. The solution involved increasing the `MaxStartups` value and exploring HAProxy's rate-limiting capabilities to gracefully handle connection spikes.