TechPrufer
Get started
Netflix
Netflix logoNetflix · Feature trail

Data Project Management

6 posts · Dec 2020 – Jun 2026

What this trail covers

Netflix is evolving its data project management by introducing 'Data Projects' as a foundational system for managing data assets at scale. This system provides a container for related data assets (tables, workflows) and a durable, synthetic identity for workloads, decoupling them from human lifecycles. It simplifies permission management by allowing grants and roles to be applied at the project level rather than per asset. The 'gravity' feature automatically associates newly created assets with

All Netflix trails
  1. The Data Canary: How Netflix Validates Catalog MetadataIntroduced the 'Data Canary' system, an automated validation mechanism for catalog metadata. This system uses a dedicated orchestrator pattern, leverages and extends the chaos platform for rapid testing (under 10 minutes), and utilizes production traffic with behavioral metrics like Starts Per Second (SPS) for validation. It incorporates sticky canaries, custom threshold tuning, and multi-tenant testing to ensure rapid and reliable detection of data corruption. The system also includes production-hardened edge case handling for redeployments and version synchronization. Controlled failure injection experiments validated its effectiveness in detecting and blocking corrupted data.
  2. Data Projects: Managing Data Assets at Netflix ScaleThis post details the introduction of 'Data Projects' to manage data assets at Netflix scale. It addresses two key problems: permissions management that struggles with organizational changes and workloads tied to fragile human identities. Data Projects introduces a container for related assets and a durable, synthetic identity for workloads, decoupling them from human lifecycles. This simplifies permission management by allowing grants and roles at the project level, and the 'gravity' feature automatically associates newly created assets with the project. The system uses a Netflix application identity and optionally an AWS IAM role for durable workload execution, improving stability and auditability.
  3. Streamlining Membership Data Engineering at Netflix with PsybergThis post introduces Psyberg, an incremental data processing framework that leverages Iceberg metadata (snapshots and partitions) to manage batch data processing, specifically addressing challenges with late-arriving data. Psyberg automates data loads, detects and manages late-arriving data without manual intervention, and integrates into data pipelining workflows, enhancing efficiency, accuracy, and timeliness. It uses Iceberg's snapshot and partition metadata to track changes and manage workloads, improving upon previous manual intervention strategies for late data.
  4. Ready-to-go sample data pipelines with DataflowThis post introduces Dataflow, a CLI utility designed to streamline data pipeline development at Netflix. It details the 'sample workflows' feature, which provides ready-to-go templates for bootstrapping batch data pipelines. The post outlines the business logic (top 100 movies/shows per country) and common components (workflow definitions, DDL, transformation jobs, audits, unit tests) used in these sample pipelines, illustrating how they can be adapted for different languages and use cases.
  5. MediumThis post introduces 'Data Projects' as a new foundational system for managing data assets at Netflix. It provides a container for related data assets (tables, workflows) and a durable, synthetic identity for workloads, decoupling them from human lifecycles. This simplifies permission management by allowing grants and roles to be applied at the project level rather than per asset. The 'gravity' feature automatically associates newly created assets with their project.
  7. Optimizing data warehouse storageThis post introduces AutoOptimize, a system designed to efficiently and transparently optimize data and metadata storage layout in data warehouses, focusing on use cases like file merging, sorting, compaction, and metadata optimization. It employs design principles of 'just-in-time' optimization, 'essential' optimization, and 'minimum replacement' to reduce resource usage and latency. The architecture is split into a decision-making Service and action-performing Actors, leveraging Apache Iceberg for table format and introducing 'Partition Entropy' for early pruning.