R2 SQL: a deep dive into our new distributed query engine

This post details the architecture and techniques behind R2 SQL, a serverless query engine for R2 object storage. It introduces the Query Planner, which uses Iceberg metadata (partition-level and column-level stats) to prune irrelevant data, and the Query Execution system, which distributes work across Cloudflare's network. Key innovations include a streaming planning pipeline that emits work units as they become available and a 'finish early' strategy enabled by ordering work units based on the query's ORDER BY clause and metadata statistics, allowing queries to complete after processing only a fraction of the data.