Accelerating LinkedIn Sales Navigator's search system with Spark transformations

This post details the optimization of LinkedIn Sales Navigator's search data manipulation pipeline, transitioning from MapReduce to Spark and significantly tuning Spark jobs. The primary focus is on reducing the total execution time of over 100 DM jobs from 6-7 hours to approximately 3 hours. Key optimization strategies include pruning the job graph by consolidating dependent jobs, identifying and optimizing critical path bottlenecks, mitigating data skewness through repartitioning based on unique search document ID, and adjusting the number of shuffle partitions. Broadcast join optimization is also mentioned as a technique for joining tables of significantly varied sizes.