TechPrufer
Get started
Yahoo
Yahoo logoYahoo · Feature trail

Apache Tez Migration and Shuffle Handler Improvement

1 post · Feb 2018

What this trail covers

Yahoo (Oath) successfully migrated its jobs from Apache MapReduce to Apache Tez, a DAG-based execution engine. A significant bottleneck in the Shuffle Handler Service was identified and addressed through community collaboration, leading to a new Shuffle Handler implementation. This resulted in a 100% migration score for Tez and the rollout of the new Shuffle Handler to research clusters, processing approximately 100 million Tez DAGs with 50 billion tasks over 38,000 nodes.

All Yahoo trails
  1. Success at Apache: A Newbie’s NarrativeThis post details the author's personal journey and contributions to the Apache Tez project, focusing on the migration of Yahoo jobs from MapReduce to Tez and the significant effort to improve the Tez Shuffle Handler Service. It highlights the challenges of the last 20% of migration, the collaborative nature of open-source development, and the specific work done on progress reporting and the new Shuffle Handler implementation, which was eventually rolled out to research clusters processing a massive scale of DAGs and tasks.