Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework

This post details the implementation of an automated schema evolution framework for Pinterest's CDC ingestion platform. It introduces a three-phase convergence model (schema divergence, code convergence, data convergence) to manage schema changes safely. The framework supports additive-only schema changes with limited type widening, uses a PR-based workflow for auditing and versioning, and employs both push-based and pull-based detection mechanisms. It also addresses unsupported changes and ambiguous CREATE TABLE diffs by leveraging an audit trail and Skeema for safe transitions.