Apache Spark @Scale: A 60 TB+ production use case

What this post added

This post details the migration of a large-scale Hive-based feature preparation pipeline for entity ranking to Apache Spark. It highlights the challenges of the previous sharded Hive implementation (3-day runtime, difficult monitoring) and the technical solutions implemented to run a single Spark job on 60 TB of compressed data with a 90 TB shuffle. Key contributions include reliability fixes for node reboots (PipedRDD robustness, configurable fetch failures, restartable shuffle service), addressing driver issues (O(N^2) operations, excessive speculation), and executor OOM fixes. Performance optimizations involved memory leak fixes in the sorter and Snappy optimizations. The post also details the tools used for performance bottleneck identification (Spark UI Metrics, Jstack, Perf/Flame Graph) and emphasizes the contribution of these improvements back to the open-source Apache Spark project.