TechPrufer
Get started
Hex
Hex logoHex · Feature trail

Lazy Dataframe Compute Engine

2 posts · Aug 2024 – Sep 2024

What this trail covers

Hex has evolved its compute engine by introducing lazy loading of dataframes using DuckDB and Arrow, significantly improving performance for multi-modal analytics. This new architecture avoids materializing large datasets into local memory for every cell, instead processing data directly via DuckDB queries against Arrow data stored in S3. This results in 5-10x speedups for project runtimes, particularly for projects with heavy SQL and no-code cell usage. The system also lays the groundwork for future optimizations by translating operations into an internal DSL (HexQL) that can be executed by various SQL dialects or pushed down to customer warehouses.

All Hex trails
  1. Optimizing Multi-Modal Analysis by Lazy Loading DataframesThis post details the migration of Hex's cell backends to a new DuckDB-based architecture that directly queries Arrow data stored remotely in S3, instead of materializing dataframes into local memory. It introduces the concept of 'LazyDF' objects which defer data loading until needed, and explains the technical implementation using DuckDB, Arrow, and S3. The post quantifies performance improvements (5-10x speedups) and discusses the implications for future development, including the translation of operations into HexQL.
  2. The future of analytics is multi-modalThis post introduces the concept of multi-modal analytics, where different data interaction modalities (SQL, Python, AI, no-code) are integrated into a single platform. It highlights Hex's 'cells' system as the foundation for this multi-modality, allowing users to combine various cell types (charts, SQL, Python, filters, pivots, etc.) within a reactive graph. The post emphasizes that this integration reduces tool-jumping and fragmentation in data workflows, enabling a more cohesive and efficient analytical experience.