Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework

This post introduces and details the 'partition finalization' mechanism within Pinterest's next-generation DB ingestion framework. It addresses the challenge of determining data completeness in a stream-based system by introducing event-time based watermarking. Key technical contributions include: 1. Collecting event time statistics (min, max, percentiles via t-digest) within Flink checkpoints and storing them in Iceberg snapshot summaries. 2. Generating a partition finalization watermark based on these statistics, ensuring it's monotonically increasing and handling late data. 3. Storing the watermark as an Iceberg table property and propagating it to downstream base tables. 4. Introducing generic extension points ('Accumulators' and 'CommitProcessors') in the Flink-to-Iceberg sink to enable this logic and other custom post-commit operations with zero overhead when unused.