Unlocking Efficiency and Performance: Navigating the Spark 3 and EMR 6 Upgrade Journey at Slack

This post details the technical migration of Slack's data workload from EMR 5 (Spark 2/Hive 2) to EMR 6 (Spark 3). Key technical contributions include: migrating the Hive Metastore from 2.3.0 to 3.1.0 using Hive Schema Tool, implementing EMR cluster provisioning via Go SDK's RunJobFlow API with custom configurations for cost control, adapting the Bazel build system for parallel Spark 2.x and 3.x JAR builds, enhancing Airflow Spark operators with a boolean flag for cluster routing and logical Spark config sizing options, addressing code compatibility issues with Spark 3 (e.g., salting for skewed joins, `spark.sql.storeAssignmentPolicy`, `spark.sql.hive.convertMetastoreParquet`), and performing exact data validation using Trino and SQL `except`/`count` queries.