Build Streaming ETL Solutions with Apache Kafka & Rail Data

What this post added

This post details building streaming ETL solutions with Apache Kafka and Confluent Platform, focusing on ingesting data from ActiveMQ and REST endpoints using Kafka Connect and CLI tools. It elaborates on data wrangling using KSQL for tasks like splitting streams, applying schemas, serializing to Avro, deriving message keys, enriching data, joining with reference information, and resolving codes. It also showcases using KSQL for real-time aggregation and the integration of Kafka Connect with PostgreSQL and Amazon S3 for advanced analytics. A specific example of building an event-driven alerting system for train delays is provided, detailing the use of KSQL to join movement data with configuration topics and Kafka Connect for pushing alerts via REST API and Telegram.