A Tale of Two Flink Autoscalers

This post details the convergence on the Apache Flink community's open-source autoscaler, moving away from an in-house solution. Key contributions include adapting the OSS autoscaler's core logic as a standalone library for Netflix's custom control plane, orchestrating scaling decisions with Temporal workflows for per-job resilience, and addressing engineering gaps in metric collection for high parallelism (up to 3,000 subtasks), preserving forward chaining by scaling connected vertices as a unit, and detecting async-sink backpressure. The post also discusses the trade-off of a lower target utilization (0.45) for improved stability and reports a 58% reduction in annualized Flink compute expenditures for the client telemetry and logging team.