Headless Data Architecture, Part 2: How to Build It

What this post added

This post introduces the concept of a headless data architecture, which formalizes a data access layer at the center of an organization using both streams (Kafka) and tables (Iceberg). It details the problems with traditional multi-hop and medallion architectures (slowness, cost, brittleness, data sprawl) and proposes a shift-left approach. The post explains how to build this architecture using data products composed of a Kafka stream and an Iceberg table, with a focus on materializing streams into tables using Kafka Connect or Confluent's Tableflow. It also outlines a modular, incremental process for selecting data sets and shifting them left, and discusses considerations for stream and table evolution management.