exa-d: Data Framework to Process the Web

What this post added

This post introduces exa-d, a new data processing framework for Exa. exa-d is designed to handle the ingestion and processing of web data at scale, addressing challenges related to numerous outputs per page, heterogeneous content, varying update frequencies, and sheer volume. Key features include typed columns and declarative dependencies, surgical updates and full rebuilds for efficient data modification, and efficient parallel execution. It uses Lance for storage on S3, with a storage model that supports writing or deleting a single column for a specific fragment without rewriting the entire fragment. The framework represents data transformations as a dependency graph and leverages this graph for execution order and to ensure column definitions act as contracts.