Accelerating LinkedIn Sales Navigator's search system with Spark transformations

What this post added

This post details the optimization of LinkedIn Sales Navigator's search data manipulation pipeline, transitioning from MapReduce to Spark and significantly tuning Spark jobs. The primary focus is on reducing the total execution time of over 100 DM jobs from 6-7 hours to approximately 3 hours. Key optimization strategies include pruning the job graph by consolidating dependent jobs, identifying and optimizing critical path bottlenecks, mitigating data skewness through repartitioning based on unique document IDs, and managing the number of shuffle partitions. The post also touches upon broadcast joins as a query processing optimization technique for tables of varied sizes.