7 Tips For Optimizing Apache Flink Applications - Shopify
Mar 8, 2022
Summary
This post details 7 tips for optimizing Apache Flink applications, focusing on profiling tools (Async-profiler, VisualVM, jemalloc+jeprof, Eclipse MAT), avoiding Kryo serialization for performance, and tuning configurations based on workload (backfill vs. steady state). It covers input source partitions, back pressure, sink throttling, networking buffers, and checkpointing strategies. It also highlights profiling heap usage with Flink's File Sink and common issues like Scala's BigDecimal and ADTs causing Kryo fallback.
More from the Apache Airflow Orchestration at Scale trail
- 3 (More) Tips for Optimizing Apache Flink Applications - Shopify
- Reducing BigQuery Costs: How We Fixed A $1 Million Query - Shopify
- Lessons Learned From Running Apache Airflow at Scale - Shopify
- Scaling Shopify's BFCM Live Map: An Apache Flink Redesign - Shopify
- Apache Beam for Search: Getting Started by Hacking Time - Shopify