Kafka Streams: Turning Data at REST to Data in Motion

What this post added

This post details how a proof of concept was built using Kafka Streams and Confluent Platform to ingest data from a REST API (flight systems) and process it into a near-real-time dashboard. It covers the architecture, including a Kafka producer for ingesting data, Kafka Streams for transformations and aggregations (e.g., flight lists, top airlines, fastest flights), and WebSockets for streaming processed data to a browser-based Progressive Web App (PWA) built with Vue.js and TypeScript. The post highlights challenges with the REST API's call limits and the decision to use Kafka Streams for its ability to handle backpressure and perform windowed aggregations. It also touches on data refinement, system resiliency, data update, and optimizing machine resources.