Detecting silent errors in the wild: Combining two novel approaches to quickly detect silent data corruptions at scale

What this post added

This post introduces and details two novel approaches, Fleetscanner (out-of-production testing) and ripple (in-production testing), for detecting silent data corruptions (SDCs) at scale within Meta's infrastructure. It compares their effectiveness, highlighting that ripple testing can detect 70% of fleet data corruptions in 15 days, while opportunistic testing takes about 6 months for the same. The post emphasizes the importance of combining both methods for comprehensive SDC detection and mitigation, detailing the infrastructure trade-offs and observed metrics for each approach.