How Meta keeps its AI hardware reliable

This post details Meta's methodologies for detecting Silent Data Corruptions (SDCs) in AI hardware, a critical challenge for reliable AI training and inference. It categorizes hardware faults into static, transient, and silent errors, highlighting the unique difficulties of SDCs due to their data dependency and subtle manifestation. The post introduces and elaborates on three novel SDC detection mechanisms: Fleetscanner (targeted micro-benchmarks), Ripple (co-located workload tests), and Hardware Sentinel (application exception analysis). It further discusses the specific implications of SDCs in AI training workloads, including NaN propagation and corrupted gradient variance, and their impact on inference workloads, leading to incorrect results. The contribution is the detailed explanation of these SDC detection and mitigation strategies specifically for AI hardware, building upon existing site reliability efforts.