Apache Airflow Orchestration at Scale

12/13/2022

This post provides three additional best practices for optimizing large stateful Apache Flink applications: 1. Setting the right parallelism by understanding task slot sharing and the relationship between task managers, slots, and parallelism values. 2. Avoiding sink bottlenecks by implementing batch writes and addressing data skew with techniques like bucketing when ideal keys are not available. 3. Using `HybridSource` to combine heterogeneous data sources, such as real-time Kafka topics and archived cloud storage, into a single logical stream for ordered processing and improved backfill times.

11/3/2022

This post details how the team reduced BigQuery costs for a new marketing tool's data pipeline by applying table clustering. The initial query processed 75 GB of data, leading to an estimated monthly cost of nearly $1 million USD. By clustering the BigQuery table on two relevant columns from the WHERE clause, the data processed was reduced to 108.3 MB per query, bringing the estimated monthly cost down to approximately $1,370.67 USD. The post also provides additional tips for reducing BigQuery costs, such as avoiding SELECT*, partitioning tables, and using table preview options instead of running queries for data exploration.

5/23/2022

This post details the technical challenges and solutions for running Apache Airflow at scale within Shopify. Key contributions include: 1. Optimizing file access by replacing GCSFuse with an NFS server synchronized with GCS, improving performance and control. 2. Mitigating database load from metadata by implementing a 28-day retention policy and a custom DAG for data cleanup. 3. Enhancing DAG traceability and control through a manifest file system for namespaces and ownership, and implementing Airflow Cluster Policies to enforce constraints on DAGs (e.g., DAG ID prefixing, celery queue restrictions, pool limitations, Kubernetes namespace restrictions). 4. Addressing load surges from DAG scheduling by introducing deterministically randomized schedule intervals for automatically generated DAGs.

3/8/2022

This post provides 7 tips for optimizing large stateful Apache Flink applications deployed in Kubernetes on GKE, leveraging RocksDB state backend and GCS for checkpoints. It details the use of profiling tools like Async-profiler, VisualVM, jemalloc+jeprof, and Eclipse MAT for debugging. It emphasizes avoiding Kryo serialization by fixing issues with Scala's BigDecimal and ADTs, leading to a 20% throughput increase. The post also discusses tuning Flink configurations based on workload (backfill vs. steady state), including input source partitions, back pressure, sink throttling, network buffers, and checkpointing strategies. Finally, it addresses heap profiling for Flink's File Sink, particularly with GCS, and how to resolve memory issues by analyzing heap dumps.

12/10/2021

This post details the redesign of Shopify's BFCM live map infrastructure using Apache Flink to address scalability issues and introduce new metrics. The previous system, Cricket (a Go-based stateful streaming service), was unable to handle the projected volume of checkout events. The new solution leverages Flink for filtering irrelevant checkout events and processing new metrics like product trends and unique shoppers. Flink was placed on the critical path, allowing Cricket to handle a reduced event volume for existing metrics. New metrics were computed by Flink and relayed through Cricket. The product trends metric involved calculating sales quantity percentage change over time using a specific formula and tumbling windows. The Flink jobs achieved 100% uptime during BFCM. A minor issue with Redis CPU load due to message fetching and the memory footprint of the new metric was resolved by dropping unnecessary Redis state.

11/19/2020

This post details the creation of Seamster, a new data pipeline workflow built on dbt and Google BigQuery, to address the limitations of the previous PySpark-based system (Starscream) for common reporting tasks. Key contributions include: modularizing data model development with a new source and model structure, implementing a Python-based unit testing framework with mock data and Great Expectations, and establishing CI pipelines for schema and DAG validation using temporary BigQuery datasets. The system aims to improve development time and thinking by favoring SQL for data modeling.

11/7/2018

This post details Shopify's migration of its Apache Kafka infrastructure from on-premises data centers to Google Cloud Platform (GCP). It outlines the three-step migration process, the decision to use Kubernetes (specifically StatefulSets and Custom Resources) over Virtual Machines for managing Kafka clusters, and best practices for containerizing Kafka and ensuring cluster availability. Key practices include node affinity, taints, inter-pod anti-affinity, persistent volumes, rack awareness, and readiness probes.