Lessons Learned From Running Apache Airflow at Scale - Shopify

This post details lessons learned from running Apache Airflow at scale at Shopify. Key contributions include: 1. Implementing an NFS server within the Kubernetes cluster for faster DAG file access, synchronized with GCS, to overcome GCSFuse bottlenecks. 2. Establishing a metadata retention policy of 28 days and implementing a DAG to delete historical data (DagRuns, TaskInstances, etc.) to manage database load. 3. Introducing a manifest file (YAML) for registering Airflow namespaces and associating DAGs with owners and source repositories for traceability. 4. Implementing a DAG policy that reads from the manifest file to enforce restrictions on DAGs, such as DAG ID prefixes, Celery queue usage, pool restrictions, and Kubernetes pod namespace access. 5. Utilizing deterministically randomized schedule intervals for automatically generated DAGs to prevent traffic surges and ensure consistent load distribution.