Scaling the Facebook data warehouse to 300 PB

What this post added

This post details the evolution of Facebook's Hive storage format from RCFile to ORCFile, driven by the need for improved storage efficiency for their 300 PB data warehouse. Key contributions include the design and implementation of adaptive column encodings (dictionary encoding for strings and integers based on observed values and character sets), optimizations for write performance (using hash maps for dictionaries, reducing dictionary sorting overhead, and intelligent skipping of encoding for subsequent stripes), and improvements to read performance through lazy column decompression and decoding leveraging index strides. These changes resulted in a 5x to 8x compression ratio improvement over RCFile and a 3x average improvement in write performance compared to open-source ORCFile.