How Facebook deals with PCIe faults to keep our data centers running reliably

This post details the methodology and tools developed to address PCIe faults in Meta's data centers. It introduces PCIcrawler for topology and error reporting, MachineChecker for hardware evaluation, and an in-house service for parsing dmesg/SELs to monitor PCIe error rates. The post explains how both uncorrected and corrected PCIe errors are handled, with a focus on establishing thresholds for corrected error rates that trigger hardware replacement. It also covers the detection and remediation of bad link speeds and widths, often resolved by reseating components. The workflow includes periodic checks, upstream/downstream analysis, and integration with configuration management systems. The system aims to automatically detect, diagnose, remediate, and repair PCIe faults, contributing to hardware health coverage and the reliable operation of millions of servers.