The Data Canary: How Netflix Validates Catalog Metadata

What this post added

Introduced the 'Data Canary' system, an automated validation mechanism for catalog metadata. This system uses a dedicated orchestrator pattern, leverages and extends the chaos platform for rapid testing (under 10 minutes), and utilizes production traffic with behavioral metrics like Starts Per Second (SPS) for validation. It incorporates sticky canaries, custom threshold tuning, and multi-tenant testing to ensure rapid and reliable detection of data corruption. The system also includes production-hardened edge case handling for redeployments and version synchronization. Controlled failure injection experiments validated its effectiveness in detecting and blocking corrupted data.