TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

Managed Airflow Platform

2 posts · Jul 2020 – Oct 2025

What this trail covers

Grab has established a dedicated team and platform to manage and provision Apache Airflow instances for various engineering teams. This platform standardizes deployment, scaling, dependency management, monitoring, and alerting for Airflow, reducing operational overhead and enabling faster adoption for teams requiring scheduling and orchestration capabilities. The platform leverages Kubernetes for isolated Airflow instances, Terraform for infrastructure management, and a common base Docker image. This post introduces a machine-learning predictive autoscaling solution for Flink applications, specifically targeting CPU provisioning for TaskManagers. It addresses the limitations of reactive autoscaling by forecasting workload based on source topic throughput and using a regression model to predict required CPU. This aims to prevent restart spikes, reactive spirals, and fluctuations, while also tackling the challenge of limited parallelism in certain connectors by focusing on vertical scaling.

All Grab trails
  1. Machine-learning predictive autoscaling for FlinkThis post introduces a machine-learning predictive autoscaling solution for Flink applications, specifically targeting CPU provisioning for TaskManagers. It addresses the limitations of reactive autoscaling by forecasting workload based on source topic throughput and using a regression model to predict required CPU. This aims to prevent restart spikes, reactive spirals, and fluctuations, while also tackling the challenge of limited parallelism in certain connectors by focusing on vertical scaling.
  2. The Journey of Deploying Apache Airflow at GrabThis post details the creation and evolution of a managed Apache Airflow platform at Grab. It outlines the transition from independently managed Airflow instances to a centralized platform to address inefficiencies. Key contributions include defining design considerations (containerization on Kubernetes, team-specific dependencies, common CI/CD, shared plugins, LDAP auth, Vault for secrets, ELK, Datadog/PagerDuty, job statistics ingestion), infrastructure management (KOPS to EKS, Terraform, Helm charts, Redis, RDS, Vault, ELK, Datadog), scaling strategies (resource and team scaling, instance categorization), Airflow image and version management (common base images, team customization, version support), deployment processes (DAG vs. instance deployments, automated base image updates, Python package management, configuration changes via Terraform/Atlantis), testing strategies (DAG validation, unit tests), and monitoring/alerting setup.