Preventing and Fixing Bad Data in Event Streams, Part 1

What this post added

This post delves into the challenges of handling 'bad data' within event streams, particularly in Apache Kafka, due to its immutable log nature. It contrasts batch processing techniques for data quality management (ETL/ELT, full refresh, incremental processing) with the complexities of streaming. Key takeaways include the reactive nature of data engineering, the reliance on deletion and reprocessing in batch, and the difficulty of applying these methods directly to event streams. The post sets the stage for discussing specific strategies for event streams in a subsequent part.