Data Projects: Managing Data Assets at Netflix Scale

What this post added

This post introduces 'Data Projects,' a new system designed to manage data assets and their associated identities at Netflix. It addresses challenges with fine-grained, per-asset access control lists (ACLs) and workloads tied to human identities. Data Projects act as containers for related assets and provide a durable, assumable identity for asynchronous and scheduled workloads. Key features include project-level grants and roles, a Netflix application identity and optional AWS IAM role for workloads, and the 'gravity' mechanism for automatic asset association. The system aims to improve stability, auditability, and resilience of data workflows in the face of organizational changes.