Data First, SLA Always

Introduced Trailblazer, a Spark Structured Streaming application for Change Data Capture (CDC) from MySQL databases using Debezium and Kafka. This addresses the limitations of periodic batch ingestion and improves scalability for large tables by streaming binlogs in real-time to the data lake. Key design principles for Trailblazer include externally managed checkpoints (using Redis), handling varying data volumes through Spark configurations, using data as a health indicator, and treating streams as ephemeral. Monitoring strategies for these streams include tracking the running:active jobs ratio, microbatch runtime, and job liveliness.