The Journey of Deploying Apache Airflow at Grab

This post details the creation and evolution of a managed Apache Airflow platform at Grab. It outlines the transition from independently managed Airflow instances to a centralized platform to address inefficiencies. Key contributions include defining design considerations (containerization on Kubernetes, team-specific dependencies, common CI/CD, shared plugins, LDAP auth, Vault for secrets, ELK, Datadog/PagerDuty, job statistics ingestion), infrastructure management (KOPS to EKS, Terraform, Helm charts, Redis, RDS, Vault, ELK, Datadog), scaling strategies (resource and team scaling, instance categorization), Airflow image and version management (common base images, team customization, version support), deployment processes (DAG vs. instance deployments, automated base image updates, Python package management, configuration changes via Terraform/Atlantis), testing strategies (DAG validation, unit tests), and monitoring/alerting setup.