Building Jetflow: a framework for flexible, performant data pipelines at Cloudflare

This post introduces Jetflow, a new internal framework built by the Cloudflare Business Intelligence team to address limitations in their existing ELT solution for petabyte-scale data ingestion. Jetflow achieves over 100x efficiency improvement in GB-s and >10x performance improvement in row ingestion rates. Key technical contributions include a modular pipeline design with Consumers, Transformers, and Loaders, the use of Apache Arrow for in-memory columnar data representation to minimize serialization overhead and GC pressure, and a data division strategy (RunInstance, Partition, Batch) for idempotency and efficient retries. It also simplifies schema management by inferring schemas from source systems and supports a wide range of data sources and targets.