Reliably Upgrading Apache Airflow at Slack’s Scale

This post details the challenges and solutions for upgrading Apache Airflow from version 1.8 to 1.10 at Slack's scale. It outlines the requirements for the upgrade (reliability, fast rollback, minimized downtime, history preservation) and discusses upgrade strategies (Red-Black vs. Big-Bang), opting for Big-Bang due to feasibility. Key technical details include database backup strategies (using replicas as new masters), a custom MySQL schema upgrade script to bypass Airflow's slower Python script, and solutions to specific issues encountered such as adhoc attribute removal, Presto operator incompatibility, UI issues with 'Mark Success', HiveServer2Hook DESCRIBE query failures, boto3 S3 list_keys behavior changes, and timezone awareness in Airflow 1.10. Lessons learned include the importance of runbooks, reducing tech debt, frequent upgrades, more dev DAGs, better monitoring/alerting, a phased task restart strategy, and internal code freezes during major upgrades.