Presto: Interacting with petabytes of data at Facebook

What this post added

Introduced Presto, a new distributed SQL query engine optimized for ad-hoc analysis at interactive speed, designed to address the limitations of Hive/MapReduce for petabyte-scale data warehouses. Presto employs an in-memory, pipelined execution model that avoids disk I/O between stages, significantly reducing query latency. It supports standard ANSI SQL and is implemented in Java with dynamic bytecode compilation for performance. Presto's architecture includes a storage abstraction with connectors for querying disparate data sources like HDFS, HBase, and custom systems. The system was deployed company-wide, scaled to 1,000 nodes, and is used by over a thousand employees, processing one petabyte daily, achieving 10x better CPU efficiency and latency than Hive/MapReduce for most queries. Future roadmap includes removing join/aggregation size restrictions, enabling output table writes, and developing a query accelerator with a new data format. Presto is also being open-sourced.