Automated Schema Evolution for DB Ingestion

3 posts · Jun 2026 – Sep 2026

What this trail covers

Pinterest's next-generation database ingestion framework, built on Kafka, Flink, Spark, and Iceberg, has evolved to address data completeness challenges in stream-based processing. This includes automated schema evolution and a new partition finalization mechanism. The partition finalization logic, integrated into the Flink-to-Iceberg sink, uses event time statistics collected via Iceberg snapshot summaries and t-digest sketches. A watermark is generated to indicate partition completeness, stored as an Iceberg table property and propagated to downstream base tables. This mechanism is extensible through 'Accumulators' for metric collection and 'CommitProcessors' for custom post-commit logic, ensuring zero overhead when not configured.

All Pinterest trails

Sep 2026Partition Finalization in Pinterest’s Next-Generation DB Ingestion FrameworkThis post introduces and details the 'partition finalization' mechanism within Pinterest's next-generation DB ingestion framework. It addresses the challenge of determining data completeness in a stream-based system by introducing event-time based watermarking. Key technical contributions include: 1. Collecting event time statistics (min, max, percentiles via t-digest) within Flink checkpoints and storing them in Iceberg snapshot summaries. 2. Generating a partition finalization watermark based on these statistics, ensuring it's monotonically increasing and handling late data. 3. Storing the watermark as an Iceberg table property and propagating it to downstream base tables. 4. Introducing generic extension points ('Accumulators' and 'CommitProcessors') in the Flink-to-Iceberg sink to enable this logic and other custom post-commit operations with zero overhead when unused.
Jun 2026Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion FrameworkThis post details the implementation of an automated schema evolution framework for Pinterest's CDC-based ingestion platform. It introduces a three-phase convergence model (schema divergence, code convergence, data convergence) to manage schema changes across Kafka, Flink, Spark, and Iceberg. The framework supports additive-only schema changes and limited type widening, with a PR-based workflow for review and auditing. It also describes push-based and pull-based detection mechanisms for schema drift and a deployment strategy that prioritizes Flink job availability through staging validation.
Jun 2026Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion FrameworkThis post details the implementation of an automated schema evolution framework for Pinterest's CDC ingestion platform. It introduces a three-phase convergence model (schema divergence, code convergence, data convergence) to manage schema changes safely. The framework supports additive-only schema changes with limited type widening, uses a PR-based workflow for auditing and versioning, and employs both push-based and pull-based detection mechanisms. It also addresses unsupported changes and ambiguous CREATE TABLE diffs by leveraging an audit trail and Skeema for safe transitions.
Advertisement