Introducing Velox: An open source unified execution engine
3/9/2023 · Pedro Pedreira, Masha Basmanova, Orri Erling
What this post added
Introduced Velox, an open-source unified execution engine designed to accelerate data management systems and streamline their development. Velox unifies common data-intensive components of data computation engines, providing a framework for consistent semantics and reducing work duplication. It integrates with existing Meta systems like Presto, Spark, and PyTorch (via TorchArrow), and has attracted over 150 contributors from the open-source community. Velox's architecture includes a generic type system, Apache Arrow-compatible columnar memory layout, vectorized expression evaluation, extensible function and operator APIs, I/O interfaces for connectors and file formats, and resource management primitives. Experimental results show significant speedups in data querying and CPU-bound queries when integrated with Presto (Prestissimo) and Spark (Gluten).