Shadow Testing for Flink Deployments

2 posts · May 2023 – Jul 2026

What this trail covers

Grab has evolved its data streaming capabilities by transitioning from Zeppelin notebooks to a shared FlinkSQL gateway for interactive querying and productionizing FlinkSQL pipelines. This new approach reduces cold start times, improves Flink version maintenance, and enhances integration with internal platforms. The system supports both individual analysts via a UI and programmatic access for internal platforms through headless APIs. The productionization tool allows for configuration-based pipeline creation and deployment, simplifying the process for users to build stream processing pipelines expressed as SQL queries.

All Grab trails

Jul 2026Enhancing Flink deployment with shadow testingIntroduced a Shadow Testing stage into the Flink deployment pipeline. This stage deploys a 'Shadow' Flink application in a separate Kubernetes namespace with distinct parameters (`isShadow=true`, prefixed names, separate Job ID). The Shadow application runs for a configured period (default 1 hour) to reach a steady state, allowing for comparison with the 'Main' application's behavior. Connector implementations are dynamically configured to use distinct consumer group IDs (Kafka), shifted Server ID ranges (Debezium), dedicated shadow Kafka clusters/S3 buckets, and prefixed metrics/logs. This process aims to eliminate downtime during deployment failures by identifying issues before the Main application is fully deployed and observed for 10 minutes.
Oct 2025Machine-learning predictive autoscaling for FlinkIntroduced a predictive autoscaler for Flink applications that uses machine learning to forecast workload and proactively adjust CPU resources. The system consists of a workload forecast model (time-series forecasting on Kafka throughput) and a resource prediction model (regression mapping Kafka throughput to Flink CPU usage). It aims to prevent reactive scaling spirals, restart spikes, and address limited parallelism constraints by enabling vertical scaling and deterministic resource allocation.
Jun 2025The complete stream processing journey on FlinkSQLThis post details the evolution from Zeppelin notebooks to a shared FlinkSQL gateway for stream processing. Key technical contributions include: replacing per-user Zeppelin clusters with a shared FlinkSQL gateway to reduce cold start times from 5 minutes to 1 minute; implementing a custom control plane with REST APIs layered on top of Flink's native REST API for session management, authentication, authorization, and orchestration; exposing headless APIs for programmatic integration with internal platforms; and developing a portal for FlinkSQL pipeline creation that allows users to define streaming logic as SQL statements, connect to internal platforms via off-the-shelf connectors, and configure parallelism and resources for automatic deployment to the Flink cluster.
May 2023Safer deployment of streaming applicationsThis post introduces a new deployment strategy for Flink applications that addresses the challenges of state consistency and safe rollbacks in stateful streaming. Key contributions include: 1. Implementing Flink's savepointing feature to capture application state before deployment, ensuring consistent starting states for new versions and enabling easier rollback. 2. Modifying the deployment process to ensure only one Flink application instance consumes Kafka messages at a time, preventing data loss and ensuring accuracy. 3. Enhancing monitoring with automated health probes and Slack notifications for deployment status. 4. Refining rollback mechanisms for configuration changes by leveraging Kubernetes' versioned ConfigMaps and for resource provision changes by using metadata annotations to store previous replica counts. The new flow is visualized in Spinnaker, reducing manual monitoring and stress for engineers.
Advertisement