Building Jetflow: a framework for flexible, performant data pipelines at Cloudflare

What this post added

This post introduces Jetflow, a new framework developed by Cloudflare's Business Intelligence team for building performant and efficient data pipelines. Jetflow addresses challenges with petabyte-scale data lakes and high-volume daily ingestion, achieving over 100x efficiency improvements in GB-s and >10x performance improvements in row ingestion rates. Key technical contributions include a modular design with Consumers, Transformers, and Loaders, support for configuration-as-code, and the use of Arrow as an in-memory columnar data format for efficient data transfer and reduced GC overhead. The framework also optimizes data processing by reading data in columnar formats and writing Parquet files directly from Arrow columns.