Filesystem Corruption and Outage Debugging

2 posts · Jun 2015 – Jul 2019

What this trail covers

GitLab has experienced and debugged filesystem corruption issues on NFS servers hosting Git repositories, leading to outages. The investigation involved analyzing kernel logs, understanding filesystem error messages, and considering recovery options. This led to the identification and resolution of a specific bug in the Linux v4.0 NFS client related to file handle staleness during renames, which was patched and backported to stable kernel versions. The experience highlights the collaborative nature of debugging complex system failures.

All GitLab trails

Jul 2019What we're doing to fix Gitaly NFS performance regressionsThis post details the debugging of performance regressions in Gitaly when using NFS, specifically addressing N+1 access patterns and the lack of memoization that was present with Rugged. It introduces the 'cat-file' cache feature in Gitaly (shipped in 11.11, enabled by default in 12.1) to recreate the memoization behavior of Rugged, improving Git object read performance. It also mentions future steps like automatically enabling Rugged for NFS instances and High Availability for Gitaly.
Nov 2018How we spent two weeks hunting an NFS bug in the Linux kernelThis post details the debugging of a 'Stale file handle' error in Git when using NFS v4.0. It describes reproducing the bug by keeping a file open while another machine renames it, and how this behavior differs between NFS v4.0 and v4.1. The investigation involved analyzing `strace` and `lsof` output to understand `git gc`'s file handling and using `tcpdump` and Wireshark to analyze NFS traffic. The root cause was identified as a bug in the Linux v4.0 NFS client related to NFS server delegations and their recall, leading to a kernel patch that was backported to stable versions.
Sep 2015GitLab.com outage on 2015-09-01This post details a specific incident of filesystem corruption on an NFS server hosting GitLab.com Git repositories, which caused a significant outage. The investigation involved analyzing `dmesg` output, realizing the error timestamps were misleading, and ultimately identifying a recurring daily error pattern that was not the root cause of the NFS slowdown. The team aborted an `fsck` and brought the service back online. Next steps include migrating data off the problematic ext4 filesystem and further investigation into the NFS slowdowns. Lessons learned include improving status updates during crises.
Jun 2015GitLab.com outage on 2015-05-29This post details a specific outage on GitLab.com caused by filesystem corruption on NFS servers. It outlines the timeline of the incident, the root causes (a faulty backup script and insufficient on-call training), and the immediate next steps taken to mitigate risk, including removing filesystem freeze steps from backups and adding a second backup strategy for SQL data. It also highlights the importance of regular operations drills for on-call engineers.
Advertisement