Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework

This post details the implementation of an automated schema evolution framework for Pinterest's CDC-based ingestion platform. It introduces a three-phase convergence model (schema divergence, code convergence, data convergence) to manage schema changes across Kafka, Flink, Spark, and Iceberg. The framework supports additive-only schema changes and limited type widening, with a PR-based workflow for review and auditing. It also describes push-based and pull-based detection mechanisms for schema drift and a deployment strategy that prioritizes Flink job availability through staging validation.