Hive – A Petabyte Scale Data Warehouse using Hadoop

This post introduces Hive, a petabyte-scale data warehousing framework built on Hadoop, developed at Facebook. It addresses the limitations of traditional RDBMS for rapidly growing data by providing a SQL-like interface over Hadoop's MapReduce. Key architectural components include a Metastore for metadata, a query compiler and execution engine that translates SQL to MapReduce jobs, and SerDe/ObjectInspectors for data format handling. Hive supports tables, partitions, bucketing, and user-defined types and functions. It also extends SQL with keywords like MAP, REDUCE, and TRANSFORM for custom computations, enabling users to plug in scripts for tasks not easily expressible in SQL, such as machine learning model generation.