Under the Hood: Scheduling MapReduce jobs more efficiently with Corona

What this post added

Introduced Corona, a new scheduling framework for MapReduce jobs that addresses limitations in Hadoop's original scheduling framework. Corona separates cluster resource management from job coordination by introducing a cluster manager and dedicated job trackers per job. It uses a push-based scheduling model to minimize latency and implements fair-share scheduling for better fairness guarantees and multi-tenant support. The deployment involved a staged rollout across the production cluster, migrating workloads team by team. Key benefits include improved slot refill times (17% improvement in production, 600ms in simulation), higher cluster utilization (over 95% vs 70%), reduced unfairness (14.3% to 3.6%), and halved job latency for a test job. Future work includes resource-based scheduling and integrating other applications like Peregrine.