The complete stream processing journey on FlinkSQL

This post details the evolution from Zeppelin notebooks to a shared FlinkSQL gateway for stream processing. Key technical contributions include: replacing per-user Zeppelin clusters with a shared FlinkSQL gateway to reduce cold start times from 5 minutes to 1 minute; implementing a custom control plane with REST APIs layered on top of Flink's native REST API for session management, authentication, authorization, and orchestration; exposing headless APIs for programmatic integration with internal platforms; and developing a portal for FlinkSQL pipeline creation that allows users to define streaming logic as SQL statements, connect to internal platforms via off-the-shelf connectors, and configure parallelism and resources for automatic deployment to the Flink cluster.