Streaming Data from the Universe with Apache Kafka | Confluent

What this post added

This post details the use of Apache Kafka for real-time data streaming in astronomical sky surveys, specifically the Zwicky Transient Facility (ZTF). It highlights the challenges of handling millions of transient object detections per night and the need for a scalable, real-time alert distribution system. The post discusses the selection of Apache Kafka for its consumer offset management and rewind capabilities, and Avro as a compact and schema-evolvable serialization format. It details the alert data pipeline, including image differencing, Avro serialization, Kafka production, and mirroring to other Kafka clusters. The system is designed to handle up to 2 million alerts per night, with each alert packet around 60 KB, resulting in up to 70 GB of data. The post also touches upon the ease of use for scientists, with Python clients and the ability to deploy custom filters in Docker containers. The system enables near real-time science by distributing alerts within approximately 20 minutes of image capture and four seconds from the main Kafka hub to external consumers.