GitLab.com outage on 2015-09-01
Sep 1, 2015Jacob Vosmaer

Summary
This post details a specific incident of GitLab.com being offline for an hour due to perceived filesystem corruption on an NFS server. It describes the initial misdiagnosis based on `dmesg` output, the decision to take the site offline for `fsck`, and the subsequent realization that the errors were recurring daily and not the cause of the immediate slowdown. The post highlights the challenges of diagnosing NFS issues, the decision to abort `fsck` and bring the site back online, and the ongoing effort to migrate data off the problematic filesystem. It also reflects on lessons learned regarding communication during crises and the need for better status updates.
More from the High Availability trail
- Consolidate your GitLab stack with Gitaly on Kubernetes
- GitLab backs 99.9% availability SLA with service credits
- Downtime happens, but GitLab Incident Management can help
- How we are closing the gap on replicating *everything* in GitLab Geo
- How we tested a feature that affected (almost) all parts of GitLab