Headless Data Architecture, Part 2: How to Build It

This post elaborates on the headless data architecture concept, building upon the previous article. It details the problems with traditional multi-hop and medallion architectures (slowness, cost, brittleness, data sprawl, trust issues) and proposes a shift-left approach. It defines a 'data product' as a combination of a Kafka stream and an Iceberg table, accessible through various 'processing heads'. The post outlines strategies for building this architecture, including creating source-to-stream workflows (using CDC or direct production) and materializing streams into Iceberg tables (via Kafka Connect or services like Tableflow). It also provides a modular, incremental approach to adopting this architecture by selecting commonly used data sets, identifying sources, creating parallel workflows, and validating the new Iceberg tables before migrating analytical jobs.