Just landed: streaming ingestion on Cloudflare with Arroyo and Pipelines

What this post added

This post announces the open beta of Cloudflare Pipelines, a new streaming ingestion product that allows users to ingest high volumes of structured, real-time data directly into R2 object storage. It also announces the acquisition of Arroyo, a cloud-native, distributed stream processing engine, to enable real-time data transformation and loading into open table formats. The post details the architecture of Pipelines, which uses Durable Objects and embedded SQLite for scalable ingestion and batching into R2, and explains how it integrates with Workers and R2 to form a data platform. It also provides examples of how to create and use a pipeline.