Orchestrating Data/ML Workflows at Scale With Netflix Maestro

This post introduces Netflix Maestro, a new data/ML workflow orchestration platform designed to overcome the scalability and usability limitations of its predecessor, Meson. Maestro is a general-purpose workflow orchestrator built as a workflow-as-a-service (WAAS). It features a core workflow engine leveraging Netflix's open-source Conductor library for state management, a time-based scheduling service, and an event-driven trigger mechanism. Key architectural components include microservices for managing workflow definitions, lifecycle, and step instances, with a focus on launching business logic in isolated containers with user identity. The platform supports complex DAG patterns, conditional branching, step templates, and parameter injection, aiming to improve operational excellence and user experience.