Migrating Slack Airflow to Python 3 Without Disruption

This post details the migration of Slack's Apache Airflow infrastructure and hundreds of DAGs from Python 2 to Python 3. The migration employed a 'red-black' deployment strategy using Celery queues, running Python 3 workers alongside Python 2 workers. Key steps included setting up a Python 3 virtual environment with pyenv and Poetry, configuring Python 3 Celery workers to listen to a 'python3' queue, patching SQLAlchemy to handle cross-version serialization issues, making DAGs Python 2/3 compatible using futurize, and gradually migrating DAGs by setting the 'queue' attribute to 'python3'. The Airflow services (web server, scheduler, flower) were also migrated to Python 3. Common issues encountered and their solutions are documented, including TypeError for bytes/str, Snowflake connector errors, dictionary ordering, file encoding issues, futurize limitations, and OpenSSL issues on macOS Catalina.