Scaling the Facebook data warehouse to 300 PB

This post details the evolution of Facebook's data warehouse storage format from RCFile to ORCFile to handle 300 PB of data. Key contributions include: the introduction of ORCFile with 256MB stripes, adaptive column encodings (dictionary encoding for strings and large integers based on heuristics like distinct value count and character set), optimizations to the ORCFile writer for improved write performance (e.g., using hash maps for dictionaries, reducing Zlib compression level), and enhancements to the ORCFile reader for lazy column decompression and decoding to speed up selective queries. These improvements resulted in an overall compression ratio increase from 5x to 8x and a 3x average improvement in write performance compared to open-source ORCFile.