How we extended Apache DataFusion to execute one query across many machines | Datadog

This post introduces Distributed DataFusion, an open-source framework built by Datadog to extend Apache DataFusion for distributed query execution. It addresses the need for scaling interactive queries across multiple machines by building on DataFusion's existing partitioning model. The framework focuses on low-latency, zero-copy streaming and efficient resource utilization. It was developed because existing distributed engines did not meet Datadog's requirements for extensibility, low-latency interactive execution, integration with Arrow/Substrait, and efficiency at scale. Benchmarks on a 12-node EC2 cluster using TPC-H and TPC-DS datasets demonstrate significant performance gains compared to Ballista, Spark, and Trino.