Looking at the code behind our three uses of Apache Hadoop

This post details Meta's use of Apache Hadoop in three key areas: data warehousing for web analytics, real-time application serving (with HBase and Scribe), and MySQL database backups. It announces the open-sourcing of their production-tuned Hadoop 0.20 branches. Key technical contributions and features highlighted include: enhancements to the NameNode's locking model for scaling to 30 petabytes, integration of Scribe for near real-time data streaming into Hadoop, and the implementation of reliable HDFS file-append operations for zero data loss with HBase. Specific innovations discussed are the AvatarNode for Hadoop High Availability (contributed as HDFS-976), HDFS-RAID with XOR-parity for storage reduction (contributed to Hadoop MapReduce), and the file-append feature for HBase (contributed to the 0.20-append branch). Future work includes cross-data center Hadoop functionality (HDFS-1432), resource-aware scheduling, slotless MapReduce, HDFS snapshots, and downtime-free JobTracker updates.