Moving an Elephant: Large Scale Hadoop Data Migration at Facebook

What this post added

This post details a large-scale data migration of dozens of petabytes of Hadoop data between data centers. It describes the decision to use a replication system over a physical move to minimize downtime. The replication process involved a bulk copy using DistCp with custom modifications for Facebook's dataset (handling multiple mappers for large files and directories with many small files) followed by a custom Hive plug-in to record changes in an audit log, which a new replication system then polled to copy modified files and metadata. The switchover involved stopping the JobTracker, allowing replication to catch up, and then updating DNS entries. Challenges included developing a multi-threaded replication system capable of handling millions of objects and managing the numerous systems interacting with the MapReduce cluster during the switchover. The replication system also demonstrated potential for disaster recovery.