Hive – A Petabyte Scale Data Warehouse using Hadoop

What this post added

This post introduces Hive, a petabyte-scale data warehousing framework built on Hadoop, developed by Facebook's Data Infrastructure Team. It addresses the limitations of previous RDBMS-based data warehouses in handling rapidly growing data volumes (from 15TB in 2007 to 2PB in 2009). Hive provides a familiar SQL-like interface and concepts (tables, columns, partitions) for querying unstructured data in Hadoop, making it accessible to users without extensive MapReduce programming knowledge. Key architectural components include a Metastore for metadata, a query compiler and execution engine that translates SQL to MapReduce jobs, and extensibility through SerDe (Serializer/Deserializer) and UDFs (User Defined Functions). The post details Hive's data model, including partitioning and bucketing for data pruning and management, and its query optimization techniques such as predicate pushdown and column pruning. It also highlights Hive's flexibility in handling various data formats and user-defined types/functions, and its support for custom Map/Reduce scripts via SQL extensions.