TechPrufer
Get started
LinkedIn
LinkedIn logoLinkedIn · Feature trail

Spark Data Pipeline Optimization

2 posts · Oct 2023 – Jul 2025

What this trail covers

This post details the optimization of LinkedIn Sales Navigator's search data manipulation pipeline, transitioning from MapReduce to Spark and significantly tuning Spark jobs. The primary focus is on reducing the total execution time of over 100 DM jobs from 6-7 hours to approximately 3 hours. Key optimization strategies include pruning the job graph by consolidating dependent jobs, identifying and optimizing critical path bottlenecks, mitigating data skewness through repartitioning based on unique search document ID, and adjusting the number of shuffle partitions. Broadcast join optimization is also mentioned as a technique for joining tables of significantly varied sizes.

All LinkedIn trails
  1. Accelerating LinkedIn Sales Navigator's search system with Spark transformationsThis post details the optimization of LinkedIn Sales Navigator's search data manipulation pipeline, transitioning from MapReduce to Spark and significantly tuning Spark jobs. The primary focus is on reducing the total execution time of over 100 DM jobs from 6-7 hours to approximately 3 hours. Key optimization strategies include pruning the job graph by consolidating dependent jobs, identifying and optimizing critical path bottlenecks, mitigating data skewness through repartitioning based on unique search document ID, and adjusting the number of shuffle partitions. Broadcast join optimization is also mentioned as a technique for joining tables of significantly varied sizes.
  2. Career stories: The math-music connection in data scienceThe author describes experimenting with sampling training data for feed recommendation algorithms, working with datasets of 500 million rows and using big data technologies like Spark and Hadoop to achieve business metric gains. Additionally, the post touches upon on-call responsibilities for the LinkedIn feed worldwide, involving immediate problem-solving for data generation pipelines to ensure member experience.