R2 SQL: a deep dive into our new distributed query engine

What this post added

This post introduces R2 SQL, a new serverless query engine that allows users to run SQL queries directly over petabytes of data stored in R2 object storage. It details the architecture, including the Query Planner that uses R2 Data Catalog (Apache Iceberg) metadata (partition-level and column-level stats) to prune irrelevant data, and the Query Execution system that leverages Cloudflare's global network and Workers for parallel processing. Key innovations include a streaming planning pipeline for early execution and a 'stop early' strategy for queries with ORDER BY clauses, enabling faster results by processing prioritized data and proving completion without scanning the entire dataset.