Blogs
Netflix logo

Netflix

Engineering blog · netflixtechblog.com

66posts0this week0watchers

Feature trails

41
Feature trailActive

Multimodal Asset Personalization

Netflix is evolving its personalization systems by incorporating multimodal embeddings (e.g., from CLIP) into asset representations. This allows models to understand the visual content of assets like artwork and video previews, significantly improving personalization for new titles (addressing the cold-start problem) and enabling model consolidation across different asset canvases. The system uses reward-based weighting for training data and inverse propensity scoring for offline evaluation to e

3 posts · 2022–2026

Timeline

20222026
Feature trailActive

Real-Time Distributed Graph Querying

Netflix is evolving its Real-Time Distributed Graph (RDG) by developing a sophisticated query serving layer. This layer is designed to efficiently handle diverse query patterns, from shallow-wide fan-outs to deep-narrow traversals, all while maintaining sub-100ms latency. Key architectural choices include a breadth-first traversal strategy, an async-first execution pipeline, selective caching of stable data, opt-in enrichments from external services, and a default to eventual consistency. This post details the ingestion and processing of data into the RDG, focusing on the challenges of handling high-volume, real-time data streams and ensuring data consistency and availability. It describes the use of Kafka for event ingestion, Flink for stream processing, and a custom data model for representing graph entities. The system employs techniques like idempotent writes and backpressure handling to manage the data flow effectively.

2 posts · 2025–2026

Timeline

20252026
Feature trailActive

Device Capability Modeling for Analytics

Netflix is building a comprehensive device capability data model and integrating feature flags to enable smarter, more granular feature management across its global device landscape. This involves designing data storage and modeling strategies for efficient analytics at scale, using cumulative and histogram tables to capture device capabilities and their distributions. This data-driven approach informs decisions about feature enablement on specific devices, ensuring performance and reliability.

1 post · 2026

Timeline

Feature trailActive

LLM Serving Infrastructure

Netflix is building a comprehensive in-house infrastructure for serving Large Language Models (LLMs) and applying them to core product features. This involves developing a full stack solution that integrates model deployment, inference, and serving within their existing production environment. Key efforts include selecting and integrating performant LLM inference engines like vLLM, managing model packaging and deployment strategies, and exposing an operational interface. GenPage represents a sig. This post details the evolution of the LLM serving platform, including the selection of vLLM as the paved-path engine, the integration of vLLM into Triton using the vLLM backend for dynamic I/O spec generation, and the adoption of an OpenAI-compatible HTTP frontend alongside gRPC for ecosystem compatibility. It also discusses deployment strategies like Red-Black and Versioned, and operational notes on boot sequences and unified metrics endpoints.

5 posts · 2022–2026

Timeline

20222026
Feature trailActive

Real-time Service Topology

Netflix is evolving its real-time service topology system to provide engineers with a unified, up-to-the-minute view of service dependencies. This system ingests data from multiple sources (eBPF network flows, IPC metrics, distributed tracing) and processes it through reactive pipelines with backpressure handling to deliver near real-time topology updates. The architecture features a multi-layer approach with physically separate topology layers (Network, IPC, Tracing) optimized for different data stores and query patterns. This post details the engineering challenges and solutions for building this system at scale, focusing on the three-stage distributed aggregation pipeline for network flow logs, backpressure handling in reactive streams, and the trade-offs involved in achieving real-time processing.

4 posts · 2020–2026

Timeline

20202026
Feature trailActive

AI Video Editing

Netflix is exploring the application of AI in video editing to streamline complex manual tasks for content creators. This involves developing models that allow for precise control over edits, preserve the integrity of original footage, and ensure natural visual continuity. Initial research focuses on layered editing (Vera) and physically plausible object removal (VOID), aiming to empower artists while maintaining creative intent. Vera, a layered video diffusion model, generates edits as separate layers, preserving original footage outside the edited regions. VOID performs physically plausible inpainting for object and interaction deletion, reconstructing the scene as if the object was never there. Both Vera and VOID are research prototypes, with papers and code released to encourage further research.

1 post · 2026

Timeline

Feature trailActive

Batch Compute Management

Netflix has evolved its batch compute management by migrating from a custom-built solution (CMB) to an open-source Kubernetes-native system, Kueue. This transition aims to simplify batch job execution, enhance resource utilization through fair sharing and preemption, and leverage the Kubernetes ecosystem for greater flexibility and innovation. The migration involved integrating Kueue into their existing container platform, Titus, and re-architecting tenant and capacity management to align with Kueue's concepts of ClusterQueues and LocalQueues. The system now manages millions of batch workloads, with improved fair sharing and preemption leading to increased resource utilization.

2 posts · 2024–2026

Timeline

20242026
Feature trailActive

Data Canary for Catalog Metadata Validation

Netflix is evolving its data validation strategies by introducing a 'Data Canary' system for catalog metadata. This system treats data deployments with the same rigor as code deployments, using production traffic to validate transformations in near real-time. It aims to detect data corruption issues in under 10 minutes, preventing bad data from reaching members and ensuring streaming reliability. The system utilizes a dedicated orchestrator pattern, extends the chaos platform with custom thresholds and sticky canaries, and focuses on behavioral metrics like Starts Per Second (SPS) for immediate abort on regression.

1 post · 2026

Timeline

Feature trailActive

Data Project Management

Netflix is evolving its data project management by introducing 'Data Projects' as a foundational system for managing data assets at scale. This system provides a container for related data assets (tables, workflows) and a durable, synthetic identity for workloads, decoupling them from human lifecycles. It simplifies permission management by allowing grants and roles to be applied at the project level rather than per asset. The 'gravity' feature automatically associates newly created assets with

6 posts · 2020–2026

Timeline

20202026
Feature trailActive

Content Launch Risk Prediction

Netflix is evolving its content launch risk prediction by developing predictive models to estimate content media asset delivery dates (Locked Cut and IMF). These models leverage production-level signals, title metadata, and seasonal data to provide more accurate and timely delivery estimates than manual schedules, aiming to reduce launch misses caused by schedule inaccuracies. The system provides both predicted and scheduled dates, allowing teams to make informed decisions and improving overall launch planning. This post details the use of boosted tree regression models and a comprehensive metrics suite for evaluation, demonstrating significant improvements in Mean Absolute Error (MAE) and Accumulated Error Days (AED) compared to manual scheduling.

1 post · 2026

Timeline

Feature trailActive

Cassandra Data Movement

Netflix has evolved its Cassandra data movement capabilities by replacing the monolithic Casspactor engine with a new layered architecture. This new system leverages a core S3 reading capability (Cassandra Analytics Wrapper) to produce standard Spark DataFrames, enabling a 'Connector Factory' model. This allows for purpose-built, data-model-aware connectors for various Cassandra data abstractions (Key Value, Time Series, etc.), improving efficiency, reducing intermediate storage bloat, enhancing performance, and enabling time travel.

2 posts · 2020–2026

Timeline

20202026
Feature trailActive

Personalized Notification System

Netflix has developed a hierarchical framework for its personalized notification system, separating long-term strategic decisions from real-time tactical execution. This involves a 'Slow' policy that creates a personalized weekly messaging plan for members, considering long-term engagement and potential fatigue, and a 'Fast' policy that handles immediate send decisions within those strategic guardrails. This decoupling allows for independent optimization of pacing strategies and message selection. This post details the evolution to this hierarchical architecture, highlighting the 'Slow' policy's role in defining personalized pacing and the 'Fast' policy's role in maximizing immediate relevance, communicating via a low-latency feature store. Key results include significant metric lifts, particularly for casual viewers, and the ability to iterate independently on pacing strategies and ranking models.

1 post · 2026

Timeline

Feature trailActive

Automating RDS PostgreSQL to Aurora PostgreSQL Migration

Netflix is evolving its database migration capabilities by developing automated processes for migrating from RDS PostgreSQL to Aurora PostgreSQL. This involves creating tools and workflows to streamline the process, reduce manual effort, and ensure data integrity and minimal downtime during the migration. The goal is to enable faster and more reliable database upgrades and modernizations.

1 post · 2026

Timeline

Feature trailCooling

Incident Management Platform

Netflix is evolving its incident management capabilities by developing a comprehensive platform to empower engineers. This platform aims to streamline incident response, improve post-mortem analysis, and foster a culture of learning and continuous improvement within the engineering organization. The focus is on providing tools and workflows that reduce the cognitive load during incidents and facilitate knowledge sharing.

2 posts · 2022–2025

Timeline

20222026
Feature trailCooling

ML Observability for Payments

Netflix is evolving its ML observability capabilities by developing comprehensive solutions for monitoring, understanding, and gaining insights into the performance and behavior of machine learning models in production, with a specific focus on payment processing. This includes implementing robust logging, monitoring, and explainability (using SHAP) to ensure reliable and transparent operation of ML-driven payment systems, reducing friction for members and enabling faster troubleshooting and continuous improvement.

1 post · 2025

Timeline

20252026
Feature trailCooling

Data/ML Workflow Orchestration Platform

Netflix is evolving its data and ML workflow orchestration capabilities by introducing Maestro, a next-generation, horizontally scalable, and user-friendly platform. Maestro replaces the previous Meson system to address scale limitations and improve usability for a diverse range of users, including data scientists, engineers, and content producers. It supports complex DAGs, event-driven triggers, and provides a workflow-as-a-service model. This post details the configurable nature of Metaflow, a popular Python library for building and deploying data science workflows, and its integration with Maestro. It highlights how Metaflow's configuration system, including its support for environment variables, JSON configuration files, and command-line arguments, allows for flexible deployment and execution across different environments. The post also touches upon the benefits of this configurability for managing complex data pipelines and ensuring reproducibility.

1 post · 2022–2024

Timeline

20222026
Feature trailQuiet since 2024

SafeTest Frontend Testing Library

Netflix is introducing SafeTest, a novel library for front-end End-to-End (E2E) testing. SafeTest aims to bridge the gap between unit testing's control over test fixtures and integration testing's control over the browser driver. It achieves this by injecting hooks during application bootstrapping, allowing for Playwright to control the browser while maintaining deep integration with the application's internal state. This enables features like direct assertion on browser spies, deep linking to tests, and flexible overrides of application dependencies and values.

1 post · 2024

Timeline

20242026
Feature trailQuiet since 2024

Microservice-based Video Processing Pipeline

Netflix is rebuilding its video processing pipeline on a next-generation microservice-based computing platform called Cosmos. This initiative aims to increase system flexibility and feature development velocity by decomposing the monolithic Reloaded platform into fine-grained, workflow-driven microservices. The Video Quality Service (VQS) is a key component of this evolution, enabling independent development and deployment of video quality algorithms and computations, such as VMAF, PSNR, and SSIM, decoupled from the encoding process. Bridging workflows manage the coexistence of Reloaded and Cosmos systems during migration, allowing for seamless operation and ubiquitous feature rollout.

2 posts · 2021–2024

Timeline

20212026
Feature trailQuiet since 2023

Scene Change Detection

Netflix is developing and deploying systems for detecting scene changes in audiovisual content. This capability is crucial for various downstream applications such as content analysis, editing, and potentially personalization. This post details the technical approach to finding cuts with smooth visual transitions, likely an evolution of the scene change detection system.

2 posts · 2023

Timeline

20232026
Feature trailQuiet since 2023

Critical Traffic Migration

Netflix is evolving its capabilities for migrating critical traffic at scale with zero downtime. This involves developing sophisticated strategies and tooling to ensure seamless transitions of high-volume, mission-critical systems, minimizing user impact and maintaining service availability throughout the migration process. The introduction of a traffic replay framework, building upon existing migration infrastructure, allows for realistic simulation of projected traffic weeks ahead of launch to

3 posts · 2023

Timeline

20232026
Feature trailQuiet since 2023

GraphQL Migration

Netflix is migrating its mobile applications from Falcor to Federated GraphQL, involving a complete overhaul from the client to the API layer. This migration is being executed in phases, with initial steps involving a GraphQL shim service and subsequent deprecation of legacy APIs in favor of domain-team-owned GraphQL services. The process emphasizes safety through techniques like A/B testing, replay testing for idempotent APIs, and sticky canaries for non-functional and non-idempotent requirements. This post details the query language and search capabilities built on top of this federated graph, leveraging Elasticsearch for indexing and a custom DSL for querying, with late-binding authorization and dynamic schema registration for efficient onboarding and customization.

2 posts · 2022–2023

Timeline

20222026
Feature trailQuiet since 2023

FileVault Key Escrow Automation

Netflix is evolving its endpoint security capabilities by open-sourcing Escrow Buddy, a tool designed to automate the remediation of missing FileVault recovery keys on managed Macs. This addresses a critical gap in device management by ensuring that security and IT teams can reliably retrieve recovery keys, preventing data loss and productivity impacts for employees locked out of their machines. The system leverages macOS authorization plugins and integrates with MDM solutions to seamlessly generate and escrow new keys without user interaction, enhancing both security posture and user experience.

1 post · 2023

Timeline

20232026
Feature trailQuiet since 2023

Native Frame Rate Playback

Netflix is evolving its playback experience by implementing native frame rate playback across various consumer streaming devices. This involves leveraging device capabilities and developing in-app solutions to mitigate visual artifacts like judder, which occur during frame rate conversions. The goal is to maximize member immersion by delivering a smooth, glitch-free viewing experience, adapting to different HDMI standards and device limitations.

1 post · 2023

Timeline

20232026
Feature trailQuiet since 2023

Scalable Annotation Service

Netflix has developed Marken, a scalable annotation service that allows any microservice to attach metadata (annotations) to their entities. Marken supports versioned annotations, real-time CRUD and search operations for UI applications, and offline analytics in Hive/Iceberg. It uses a flexible schema-based approach to define annotation data models, supporting various data types including geometry (WKT) and temporal information. The architecture leverages Cassandra for data storage and Elasticsearch for search capabilities, designed to handle high RPS and large data volumes.

1 post · 2023

Timeline

20232026
Feature trailQuiet since 2023

Causal Machine Learning for Creative Insights

Netflix is developing and applying causal machine learning techniques to gain deeper insights into creative decision-making and content performance. This involves moving beyond correlational analysis to understand the true impact of creative choices on member engagement and satisfaction. The system aims to provide actionable insights that can inform content strategy and production, ultimately leading to more successful content launches and improved member experience. This post surveys various ap

3 posts · 2021–2023

Timeline

20212026
Feature trailQuiet since 2023

Device Management Platform

Netflix is building a reliable Device Management Platform to support automated testing of hundreds of device types and partner device onboarding. The platform ingests event-sourced device updates via MQTT and Kafka, processing them into a scalable CockroachDB datastore. Key requirements include back-pressure support, in-order processing, and fault tolerance to handle dynamic traffic loads and ensure up-to-date device information for testing and certification.

1 post · 2023

Timeline

20232026
Feature trailQuiet since 2022

Neural Network Video Quality Enhancement

Netflix is enhancing video quality by leveraging neural networks to improve the visual fidelity of its content. This involves developing and deploying advanced models that can intelligently upscale, denoise, and enhance video streams, leading to a more immersive and visually appealing experience for members. The system focuses on applying these techniques to specific aspects of video processing to achieve significant quality improvements.

1 post · 2022

Timeline

20222026
Feature trailQuiet since 2022

Titus Gateway Caching

Netflix is evolving its Titus Gateway caching mechanism to improve performance and reduce latency for critical services. This involves implementing a consistent caching strategy within the gateway to ensure data freshness and efficient retrieval.

1 post · 2022

Timeline

20222026
Feature trailQuiet since 2022

Unreal Engine Validation Framework

Netflix is building a validation framework as a plugin for Unreal Engine to ensure consistent and high-quality configurations for virtual production workflows. This framework automates checks and fixes for technical issues, acting as an assistant, teacher, and enabler for filmmakers. It is extensible, customizable, and can be integrated into CI/CD pipelines or used directly within the editor. The framework supports project and level-specific validations, organized into workflows, and generates reports for archival and sharing.

1 post · 2022

Timeline

20222026
Feature trailQuiet since 2022

Out-of-Memory Kill Prediction

Netflix is evolving its mobile application's reliability by formulating out-of-memory (OOM) kill prediction as a machine learning problem. This involves developing models to proactively identify and mitigate conditions that lead to OOM kills, thereby improving app stability and user experience.

1 post · 2022

Timeline

20222026
Feature trailQuiet since 2022

Application Security Program Evolution

Netflix is evolving its application security program by refining its organizational structure and investment strategies. The program has moved from a shared on-call model for operational services to a more specialized approach. This includes productizing security for the studio ecosystem, standardizing security baselines, and building self-service capabilities. The team is also investing in dedicated functions for "Appsec Reviews and Assessments" to handle critical services that were previously managed via a shared rotation. This evolution aims to improve consistency, scalability, and provide more context-aware, opinionated automated guidance to engineering teams.

1 post · 2022

Timeline

20222026
Feature trailQuiet since 2022

Performance Regression Detection

Netflix is evolving its performance monitoring and regression detection capabilities by implementing a system that proactively identifies and addresses performance regressions before they impact users. This involves analyzing various metrics and system behaviors to catch issues early in the development lifecycle.

1 post · 2022

Timeline

20222026
Feature trailQuiet since 2022

Experimentation

Netflix is evolving its experimentation capabilities by detailing the statistical concepts of false negatives and statistical power in A/B testing. This post explains how to increase power through effect size, sample size, and reducing metric variability, building upon previous discussions of false positives and statistical significance. The goal is to improve the accuracy and reliability of A/B test interpretations to drive informed product decisions.

3 posts · 2021–2022

Timeline

20212026
Feature trailQuiet since 2021

Micro Frontend Framework

Netflix is evolving its approach to building internal tools by developing Lattice, a pluggable framework for micro frontends. This framework aims to enable low-friction adoption, weak dependencies, and rapid development by allowing applications to extend core functionality using familiar React paradigms and referencing remote bundles via Webpack Module Federation or native JavaScript Modules. Lattice provides a mechanism for hosts to expose mount points and state that plugins can manipulate or override, facilitating the creation of highly aligned yet loosely coupled micro frontend architectures.

1 post · 2021

Timeline

20212026
Feature trailQuiet since 2021

API Mutation Design with FieldMask

Netflix is evolving its API design practices by standardizing the use of Protobuf FieldMask for mutation operations (update and remove). This approach aims to simplify API maintenance, prevent API explosion from numerous field-specific endpoints, and allow for atomic updates of multiple fields or removal of specific fields within a single RPC. The system also addresses schema evolution challenges and the implications of empty or missing FieldMasks.

2 posts · 2021

Timeline

20212026
Feature trailQuiet since 2021

Application Security Gateway

Netflix is evolving its application security by productizing security controls into a gateway, aiming to streamline the process of making applications internet-facing and ensuring strong authentication as an architectural property. This involves integrating features like WAF, DDoS prevention, security header validation, and logging into a centralized system, significantly reducing the security checklist for development teams and automating infrastructure setup.

1 post · 2021

Timeline

20212026
Feature trailQuiet since 2021

Container Security with User Namespaces

Netflix is evolving its container security by leveraging Linux user namespaces on Titus agents. This enhances security isolation in multi-tenant clusters by mapping container UIDs to host UIDs, preventing unmapped UIDs from accessing host resources and allowing containers to exercise privileges within their own namespace without granting them to the host's init user namespace. This mitigates risks associated with container escapes and lateral movement.

1 post · 2021

Timeline

20212026
Feature trailQuiet since 2021

Revenue & Growth Tooling Platform

Netflix is evolving its Revenue & Growth Tools (RGT) team's capabilities by developing a comprehensive platform for building full-stack tools that support financial operations. This includes creating configuration management systems like Haze, which uses GraphQL and JSON to orchestrate backend API calls for managing configuration data, and self-service workflow tools like RunScript, which allows engineers to create secure UIs for operational users to execute common workflows using Kotlin or Java. The vision is to provide building blocks for automation, federate domain APIs via GraphQL, and act as a 'Local Central Team' to bridge platform and product teams.

1 post · 2021

Timeline

20212026
Feature trailQuiet since 2021

xHE-AAC Audio Streaming

Netflix is evolving its audio streaming capabilities by implementing Extended HE-AAC with MPEG-D DRC (xHE-AAC) for compatible Android mobile devices. This technology improves intelligibility in noisy environments, adapts to variable cellular connections, and scales to studio-quality audio. The system leverages MPEG-D DRC metadata for loudness management, ensuring consistent dialogue levels across all content, and for dynamic range control, optimizing the audio experience for various devices and environments. This builds upon previous efforts in adaptive bitrate audio and Dolby Atmos.

1 post · 2021

Timeline

20212026
Feature trailQuiet since 2020

Simple Streaming Telemetry

Netflix is evolving its streaming telemetry by introducing a simpler, more efficient system for collecting and processing real-time data. This new approach aims to reduce the overhead of traditional telemetry, improve data accuracy, and enable faster insights into streaming performance and member experience. The system focuses on collecting key metrics directly from the client and processing them with minimal latency.

1 post · 2020

Timeline

20202026