GitLab.com outage on 2015-05-29

What this post added

This post details a specific outage on GitLab.com due to an infrastructure upgrade that moved to a clustered setup. The root cause was identified as a backup script hanging during the filesystem freeze/unfreeze process on the backend server, which prevented workers from mounting the NFS share. The extended downtime was attributed to insufficient training and documentation for on-call engineers regarding the new infrastructure. Next steps included removing the filesystem freeze from the backup script, implementing a secondary backup strategy for SQL data, and initiating regular operations drills for on-call engineers.