TechPrufer
Sign inGet started

Blogs›Yahoo›Apache Tez Migration and Shuffle Handler Improvement

Apache Tez Migration and Shuffle Handler Improvement

Apache Tez Migration and Shuffle Handler Improvement

1
posts
2018

Yahoo (Oath) successfully migrated its jobs from Apache MapReduce to Apache Tez, a DAG-based execution engine. A significant bottleneck in the Shuffle Handler Service was identified and addressed through community collaboration, leading to a new Shuffle Handler implementation. This resulted in a 100% migration score for Tez and the rollout of the new Shuffle Handler to research clusters, processing approximately 100 million Tez DAGs with 50 billion tasks over 38,000 nodes.

2018

Success at Apache: A Newbie’s Narrative

2/5/2018

This post details the author's personal journey contributing to the Apache Tez project, focusing on the migration of Yahoo jobs from MapReduce to Tez and the subsequent improvement of the Tez Shuffle Handler Service. It highlights the collaborative nature of open-source development, the challenges faced in implementing progress reporting and optimizing the shuffle phase, and the eventual success of a new Shuffle Handler implementation that addressed performance bottlenecks.