TechPrufer
Sign inGet started

Blogs›LinkedIn›Spark Data Pipeline Optimization

Spark Data Pipeline Optimization

Spark Data Pipeline Optimization

2
posts
2023–2025

This post details the optimization of LinkedIn Sales Navigator's search data manipulation pipeline, transitioning from MapReduce to Spark and significantly tuning Spark jobs. The primary focus is on reducing the total execution time of over 100 DM jobs from 6-7 hours to approximately 3 hours. Key optimization strategies include pruning the job graph by consolidating dependent jobs, identifying and optimizing critical path bottlenecks, mitigating data skewness through repartitioning based on unique values, and optimizing data serialization formats. The author also describes experimenting with sampling training data for feed recommendation algorithms, working with datasets of 500 million rows and using big data technologies like Spark and Hadoop to achieve business metric gains. Additionally, the post touches upon on-call responsibilities for the LinkedIn feed worldwide, involving immediate problem-solving for data generation pipelines to ensure member experience.

2025

Accelerating LinkedIn Sales Navigator's search system with Spark transformations

7/23/2025

This post details the optimization of LinkedIn Sales Navigator's search data manipulation pipeline, transitioning from MapReduce to Spark and significantly tuning Spark jobs. The primary focus is on reducing the total execution time of over 100 DM jobs from 6-7 hours to approximately 3 hours. Key optimization strategies include pruning the job graph by consolidating dependent jobs, identifying and optimizing critical path bottlenecks, mitigating data skewness through repartitioning based on unique document IDs, and managing the number of shuffle partitions. The post also touches upon broadcast joins as a query processing optimization technique for tables of varied sizes.

2023

Career stories: The math-music connection in data science

10/2/2023

The author describes experimenting with sampling training data for feed recommendation algorithms, working with datasets of 500 million rows and using big data technologies like Spark and Hadoop to achieve business metric gains. Additionally, the post touches upon on-call responsibilities for the LinkedIn feed worldwide, involving immediate problem-solving for data generation pipelines to ensure member experience.