Rethinking Stream Processing: Data Exploration

This post introduces the adoption of Apache Zeppelin as a web-based notebook environment to enable data exploration on online data (streaming data from Kafka). It details the problem of underutilized streaming data due to exploration difficulties, the solution of using Zeppelin with Flink interpreters for SQL-based querying and visualization, and the need for dynamic schema derivation from Protobuf to DDL. The post also touches upon security requirements for data access authorization and audit trails.