TechPrufer
Get started
Slack
Slack logoSlack · Feature trail

Spark and EMR Upgrade Journey

7 posts · Dec 2016 – May 2026

What this trail covers

This thread tracks the evolution of Slack's data workload migration and performance optimization efforts, specifically focusing on upgrades to Spark and AWS EMR. Initial posts detail the challenges and strategies involved in migrating from older versions (EMR 5/Spark 2) to newer ones (EMR 6/Spark 3). This post details the migration of over 60 EMR clusters and thousands of Airflow DAGs from EMR 5.3x (Spark 2.x/Hive 2.x) to EMR 6.x (Spark 3.x), including Hive Metastore upgrade, EMR cluster provisioning, and the elimination of SSH-based job submission by migrating to a REST-based architecture using YARN Distributed Shell and Slack's Quarry gateway. This migration significantly improves security, reliability, and enables further infrastructure modernization.

All Slack trails
  1. From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data PipelinesThis post details the successful migration of over 700 SSH-based Airflow operators orchestrating critical data pipelines on EMR clusters to a REST-based architecture. The core of this migration involved eliminating direct SSH access to EMR master nodes by leveraging YARN Distributed Shell to run arbitrary shell commands within YARN containers. This was facilitated by Slack's Quarry gateway, which provides a unified REST API for job submission across compute engines. The migration was executed incrementally across 8 data regions with zero downtime for business-critical services, addressing security risks, operational pain points, and unblocking further infrastructure modernization initiatives like Spark on Kubernetes and EMR on EKS.
  2. Unlocking Efficiency and Performance: Navigating the Spark 3 and EMR 6 Upgrade Journey at SlackThis post details the technical migration of Slack's data workload from EMR 5 (Spark 2/Hive 2) to EMR 6 (Spark 3). Key technical contributions include: migrating the Hive Metastore from 2.3.0 to 3.1.0 using Hive Schema Tool, implementing EMR cluster provisioning via Go SDK's RunJobFlow API with custom configurations for cost control, adapting the Bazel build system for parallel Spark 2.x and 3.x JAR builds, enhancing Airflow Spark operators with a boolean flag for cluster routing and logical Spark config sizing options, addressing code compatibility issues with Spark 3 (e.g., salting for skewed joins, `spark.sql.storeAssignmentPolicy`, `spark.sql.hive.convertMetastoreParquet`), and performing exact data validation using Trino and SQL `except`/`count` queries.
  3. Data Lineage at SlackThis post details the implementation of data lineage ingestion for Airflow callbacks and dashboard usage. It describes the construction of a central Data Lineage Service that exposes endpoints for ingestion, parsing, and persistence of lineage data into RDS MySQL and subsequently flattened into Hive. The post highlights the use of ANTLR for SQL parsing and Pregel for building lineage graphs in Spark, enabling efficient querying and notification of downstream consumers via an internal Data Portal. Future work includes extending lineage to non-SQL Spark jobs, integrating with graph databases, and improving schema change automation.
  4. Migrating Slack Airflow to Python 3 Without DisruptionThis post details the migration of Slack's Apache Airflow infrastructure and hundreds of DAGs from Python 2 to Python 3. The migration employed a 'red-black' deployment strategy using Celery queues, running Python 3 workers alongside Python 2 workers. Key steps included setting up a Python 3 virtual environment with pyenv and Poetry, configuring Python 3 Celery workers to listen to a 'python3' queue, patching SQLAlchemy to handle cross-version serialization issues, making DAGs Python 2/3 compatible using futurize, and gradually migrating DAGs by setting the 'queue' attribute to 'python3'. The Airflow services (web server, scheduler, flower) were also migrated to Python 3. Common issues encountered and their solutions are documented, including TypeError for bytes/str, Snowflake connector errors, dictionary ordering, file encoding issues, futurize limitations, and OpenSSL issues on macOS Catalina.
  5. Reliably Upgrading Apache Airflow at Slack’s ScaleThis post details the challenges and solutions for upgrading Apache Airflow from version 1.8 to 1.10 at Slack's scale. It outlines the requirements for the upgrade (reliability, fast rollback, minimized downtime, history preservation) and discusses upgrade strategies (Red-Black vs. Big-Bang), opting for Big-Bang due to feasibility. Key technical details include database backup strategies (using replicas as new masters), a custom MySQL schema upgrade script to bypass Airflow's slower Python script, and solutions to specific issues encountered such as adhoc attribute removal, Presto operator incompatibility, UI issues with 'Mark Success', HiveServer2Hook DESCRIBE query failures, boto3 S3 list_keys behavior changes, and timezone awareness in Airflow 1.10. Lessons learned include the importance of runbooks, reducing tech debt, frequent upgrades, more dev DAGs, better monitoring/alerting, a phased task restart strategy, and internal code freezes during major upgrades.
  7. Data Wrangling at SlackThis post details the challenges of data wrangling at Slack, specifically focusing on the interoperability issues between Hive, Presto, and Spark when processing data stored in Parquet format on S3. It highlights problems with different Parquet library versions leading to data absence, schema evolution troubles (column order mismatches), and difficulties during EMR upgrades. To address these, Slack developed custom Hive InputFormat and Parquet OutputFormat to control serialization and deserialization, ensuring data consistency across engines and enabling safer upgrades.