Inside Tableflow GA: Real-Time Kafka to Iceberg

This post announces the General Availability (GA) of Tableflow, a product that materializes Apache Kafka topics into Apache Iceberg tables. It details the technical architecture of Tableflow, which leverages Kora's storage layer to convert Kafka segments into Parquet files. A new metadata publishing service uses Schema Registry to generate Iceberg metadata transaction logs, handling schema mapping, evolution, and type conversions. Key features highlighted include data conversion (Avro, JSON, Protobuf to Iceberg/Parquet), automatic schema evolution, catalog syncing (AWS Glue, Snowflake Open Catalog, Apache Polaris, Unity Catalog), and table maintenance (compaction, snapshot expiration). The post also explains Snowflake Open Catalog's role in managing Iceberg tables and provides a walk-through of the integration process, emphasizing the use of AWS AssumeRole for provider integration and external volumes/catalog integrations within Snowflake to access the Iceberg data stored in S3.