Data Projects: Managing Data Assets at Netflix Scale

This post details the introduction of 'Data Projects' to manage data assets at Netflix scale. It addresses two key problems: permissions management that struggles with organizational changes and workloads tied to fragile human identities. Data Projects introduces a container for related assets and a durable, synthetic identity for workloads, decoupling them from human lifecycles. This simplifies permission management by allowing grants and roles at the project level, and the 'gravity' feature automatically associates newly created assets with the project. The system uses a Netflix application identity and optionally an AWS IAM role for durable workload execution, improving stability and auditability.