Blogs›GitLab›Filesystem Corruption and Outage Debugging

Filesystem Corruption and Outage Debugging

posts

2015–2018

GitLab has experienced and debugged filesystem corruption issues on NFS servers hosting Git repositories, leading to outages. The investigation involved analyzing kernel logs, understanding filesystem error messages, and considering recovery options. This led to the identification and resolution of a specific bug in the Linux v4.0 NFS client related to file handle staleness during renames, which was patched and backported to stable kernel versions. The experience highlights the collaborative nature of open-source debugging.

2018

How we spent two weeks hunting an NFS bug in the Linux kernel

11/14/2018

This post details the debugging process for a 'Stale file handle' error encountered with NFS, specifically on Linux v4.0 clients. It describes reproducing the bug using a controlled experiment involving file renames and open file handles, and analyzing NFS traffic with tcpdump and Wireshark to compare NFS v4.0 and v4.1 behavior. The investigation revealed a potential issue with NFS server delegations and the handling of file handle staleness during renames, leading to a kernel patch.

2015

GitLab.com outage on 2015-09-01

9/1/2015

This post details the debugging process for a GitLab.com outage caused by apparent filesystem corruption on an NFS server. It highlights the initial misinterpretation of `dmesg` output, the decision to take the system offline for `fsck`, and the subsequent realization that the errors were recurring daily and not the immediate cause of the NFS slowdown. The post also mentions the plan to migrate data off the affected ext4 filesystem and run `git fsck` on repositories. The root cause of the NFS slowdown remains unknown.