Stream, Analyze, and Visualize Wi-Fi Data with Kafka and Friends

This post details the process of capturing Wi-Fi packet data (pcap) using a Raspberry Pi and streaming it to Apache Kafka. It demonstrates how to use Kafka Connect with Elasticsearch and Kibana for visual analysis of the data, including record counts, MAC address distributions, and packet type frequencies. It also showcases the use of ksqlDB for numerical analysis, including creating streams and tables to aggregate and query packet subtypes by day. The post further illustrates how to use ksqlDB to create a sink connector for PostgreSQL to persist these aggregates and how to use Kibana's graph capabilities to visualize relationships between devices and access points. Finally, it describes splitting the pcap data into separate Kafka topics using ksqlDB for more targeted analysis and schema management.