Just landed: streaming ingestion on Cloudflare with Arroyo and Pipelines

This post announces the open beta of Cloudflare Pipelines, a streaming ingestion product that allows high-volume, structured, real-time data ingestion directly into R2 object storage. It also announces the acquisition of Arroyo, a cloud-native stream processing engine, to enable real-time data transformation, aggregation, and SQL-based querying on ingested data. Pipelines is built using Durable Objects and embedded SQLite for immediate persistence and replication via the Storage Relay Service (SRS), enabling horizontal scalability and high availability. Data is batched, compressed, and written to R2 in Hive-partitioned, newline-delimited JSON (NDJSON) files with ULID filenames. The post details the architecture of Pipelines, including the separation of control plane and data path, and the flow of records from ingestion to R2.