Flink Autoscaling

Netflix is evolving its Flink autoscaling strategy by converging on the Apache Flink community's open-source autoscaler. This transition involves adapting the OSS autoscaler to Netflix's internal platform, which runs on a custom control plane rather than the standard Kubernetes Operator. Key adaptations include integrating the autoscaler's core logic as a standalone library, orchestrating workflows with Temporal for per-job resilience, and addressing engineering gaps in metric collection for high parallelism, preserving forward chaining, and respecting sink limits. The OSS-based autoscaler has achieved general availability for custom jobs, yielding significant compute expenditure reductions and improved stability.