Detecting silent errors in the wild: Combining two novel approaches to quickly detect silent data corruptions at scale

This post introduces and evaluates two novel approaches for detecting silent data corruptions (SDCs) at scale: opportunistic testing (Fleetscanner) and in-production testing (ripple testing). Opportunistic testing leverages planned maintenance events for longer, more intrusive tests, while ripple testing integrates brief, performance-aware tests alongside production workloads for faster detection of certain error types. The post demonstrates that combining both methods provides comprehensive coverage, with ripple testing detecting 70% of fleet corruptions in 15 days, and opportunistic testing covering the remaining 23% over 6 months. It highlights the trade-offs and benefits of each approach, emphasizing the need for both to ensure fleet resiliency and prevent data loss.