Lessons Learned From Running Apache Airflow at Scale - Shopify

What this post added

This post details the technical challenges and solutions for running Apache Airflow at scale within Shopify. Key contributions include: 1. Optimizing file access by replacing GCSFuse with an NFS server synchronized with GCS, improving performance and control. 2. Mitigating database load from metadata by implementing a 28-day retention policy and a custom DAG for data cleanup. 3. Enhancing DAG traceability and control through a manifest file system for namespaces and ownership, and implementing Airflow Cluster Policies to enforce constraints on DAGs (e.g., DAG ID prefixing, celery queue restrictions, pool limitations, Kubernetes namespace restrictions). 4. Addressing load surges from DAG scheduling by introducing deterministically randomized schedule intervals for automatically generated DAGs.