TechPrufer
Sign inGet started

Blogs›Netflix›Data Project Management

Data Project Management

Data Project Management

2
posts
2026

Netflix has introduced 'Data Projects' as a foundational system for managing data assets at scale. This system provides a container for related data assets (tables, workflows) and a durable, synthetic identity for workloads, decoupling them from human lifecycles. It simplifies permission management by allowing grants and roles to be applied at the project level rather than per asset. The 'gravity' feature automatically associates newly created assets with their parent project, ensuring organization. This post introduces the 'Data Canary' system, an automated approach to validate catalog metadata transformations using production traffic, ensuring data integrity and reliability by detecting issues in under 10 minutes and blocking bad data from reaching members. This extends the concept of validating data assets to ensuring the quality of the data itself within the broader data management framework.

2026

The Data Canary: How Netflix Validates Catalog Metadata

6/20/2026

Introduced the 'Data Canary' system, an automated validation mechanism for catalog metadata. This system uses a dedicated orchestrator pattern, leverages and extends the chaos platform for rapid testing (under 10 minutes), and utilizes production traffic with behavioral metrics like Starts Per Second (SPS) for validation. It incorporates sticky canaries, custom threshold tuning, and multi-tenant testing to ensure rapid and reliable detection of data corruption. The system also includes production-hardened edge case handling for redeployments and version synchronization. Controlled failure injection experiments validated its effectiveness in detecting and blocking corrupted data.

Data Projects: Managing Data Assets at Netflix Scale

6/20/2026

This post introduces 'Data Projects,' a new system designed to manage data assets and their associated identities at Netflix. It addresses challenges with fine-grained, per-asset access control lists (ACLs) and workloads tied to human identities. Data Projects act as containers for related assets and provide a durable, assumable identity for asynchronous and scheduled workloads. Key features include project-level grants and roles, a Netflix application identity and optional AWS IAM role for workloads, and the 'gravity' mechanism for automatic asset association. The system aims to improve stability, auditability, and resilience of data workflows in the face of organizational changes.