Composable data management at Meta

This post details Meta's shift towards a composable data management architecture, moving away from fragmented monolithic systems. It introduces the concept of factoring out common logical components (language frontend, IR, query optimizer, execution engine, execution runtime) as reusable libraries. The core contribution is the development and open-sourcing of Velox, a unified, engine- and dialect-agnostic execution engine that leverages columnar and vectorized execution, compressed execution, lazy materialization, and adaptivity. The post also emphasizes the importance of open standards like Apache Arrow for interoperability and discusses the ongoing work in system-wide integration and extensibility for domain-specific features.