Saving capacity with HDFS RAID

This post details the implementation and lessons learned from deploying HDFS RAID (Erasure Codes) to reduce HDFS replication factors, leading to tens of petabytes of capacity savings. It introduces XOR and Reed-Solomon erasure codes, explains the 'small file problem' and the solution of 'directory RAID', and discusses challenges like data corruption, parallelizing RAID for large directories, and handling directory changes. Future work includes native HDFS RAID support.