Data Wrangling at Slack

This post details the challenges of data wrangling at Slack, specifically focusing on the interoperability issues between Hive, Presto, and Spark when processing data stored in Parquet format on S3. It highlights problems with different Parquet library versions leading to data absence, schema evolution troubles (column order mismatches), and difficulties during EMR upgrades. To address these, Slack developed custom Hive InputFormat and Parquet OutputFormat to control serialization and deserialization, ensuring data consistency across engines and enabling safer upgrades.