Bulldozer: Batch Data Moving from Data Warehouse to Online Key-Value Stores

This post introduces Bulldozer, a self-serve data platform for batch data movement from data warehouses to online key-value stores. Bulldozer leverages Spark for data processing and a Key-Value Data Abstraction Layer (KV DAL) to decouple applications from specific datastore APIs. It supports configuration-based job definitions via YAML, auto-generates Protobuf schemas for serialization, and ensures data integrity and seamless transitions for consumers. The system handles version control of datasets, ensuring atomic writes and seamless reads of new data versions.