3 (More) Tips for Optimizing Apache Flink Applications - Shopify

This post provides three additional best practices for optimizing large stateful Apache Flink applications: 1. Setting the Right Parallelism: Discusses configuring parallelism at operator, execution environment, client, and system levels, recommending starting with execution environment level parallelism and leveraging task slot sharing. It provides a rule of thumb for calculating the required number of task managers based on parallelism and slot count. 2. Avoiding Sink Bottlenecks: Explains how data sinks can become bottlenecks, leading to backpressure. It recommends batch writing to sinks for improved throughput and efficiency, and addresses data skew by suggesting bucketing techniques when ideal keys are not available. 3. Using HybridSource to Combine Heterogeneous Sources: Introduces HybridSource as a mechanism to abstract multiple heterogeneous data sources into a single logical source, enabling ordered processing. It uses the example of combining Kafka archives with real-time Kafka topics to provide a complete historical data view, highlighting the benefits of superior object storage partitioning for faster backfill times.