Apache Airflow Orchestration at Scale

7 posts · Nov 2018 – Dec 2022

What this trail covers

Shopify has evolved its data pipeline platform from PySpark (Starscream) to a more efficient SQL-based workflow using dbt (Seamster) for reporting and common data tasks. This involved modularizing data model development, implementing unit testing with Great Expectations, and providing schema and DAG validation through temporary BigQuery datasets to reduce development time and improve the consistency of business metrics. The platform has also migrated its Kafka infrastructure to Kubernetes, enabling reliable cluster deployment with predictable and scalable performance, and has adopted best practices for containerizing Kafka and managing cluster availability.

All Shopify trails

Dec 20223 (More) Tips for Optimizing Apache Flink Applications - ShopifyThis post provides three additional best practices for optimizing large stateful Apache Flink applications: 1. Setting the Right Parallelism: Discusses configuring parallelism at operator, execution environment, client, and system levels, recommending starting with execution environment level parallelism and leveraging task slot sharing. It provides a rule of thumb for calculating the required number of task managers based on parallelism and slot count. 2. Avoiding Sink Bottlenecks: Explains how data sinks can become bottlenecks, leading to backpressure. It recommends batch writing to sinks for improved throughput and efficiency, and addresses data skew by suggesting bucketing techniques when ideal keys are not available. 3. Using HybridSource to Combine Heterogeneous Sources: Introduces HybridSource as a mechanism to abstract multiple heterogeneous data sources into a single logical source, enabling ordered processing. It uses the example of combining Kafka archives with real-time Kafka topics to provide a complete historical data view, highlighting the benefits of superior object storage partitioning for faster backfill times.
Nov 2022Reducing BigQuery Costs: How We Fixed A $1 Million Query - ShopifyThis post details how the team reduced BigQuery costs for a new marketing tool's data pipeline by applying table clustering. The initial query processed 75 GB of data, leading to an estimated monthly cost of nearly $1 million USD. By clustering the BigQuery table on two relevant columns from the WHERE clause, the data processed was reduced to 108.3 MB per query, bringing the estimated monthly cost down to approximately $1,370.67 USD. The post also provides additional tips for reducing BigQuery costs, such as avoiding SELECT*, partitioning tables, and using table preview options instead of running queries for data exploration.
May 2022Lessons Learned From Running Apache Airflow at Scale - ShopifyThis post details lessons learned from running Apache Airflow at scale at Shopify. Key contributions include: 1. Implementing an NFS server within the Kubernetes cluster for faster DAG file access, synchronized with GCS, to overcome GCSFuse bottlenecks. 2. Establishing a metadata retention policy of 28 days and implementing a DAG to delete historical data (DagRuns, TaskInstances, etc.) to manage database load. 3. Introducing a manifest file (YAML) for registering Airflow namespaces and associating DAGs with owners and source repositories for traceability. 4. Implementing a DAG policy that reads from the manifest file to enforce restrictions on DAGs, such as DAG ID prefixes, Celery queue usage, pool restrictions, and Kubernetes pod namespace access. 5. Utilizing deterministically randomized schedule intervals for automatically generated DAGs to prevent traffic surges and ensure consistent load distribution.
Mar 20227 Tips For Optimizing Apache Flink Applications - ShopifyThis post details 7 tips for optimizing Apache Flink applications, focusing on profiling tools (Async-profiler, VisualVM, jemalloc+jeprof, Eclipse MAT), avoiding Kryo serialization for performance, and tuning configurations based on workload (backfill vs. steady state). It covers input source partitions, back pressure, sink throttling, networking buffers, and checkpointing strategies. It also highlights profiling heap usage with Flink's File Sink and common issues like Scala's BigDecimal and ADTs causing Kryo fallback.
Dec 2021Scaling Shopify's BFCM Live Map: An Apache Flink Redesign - ShopifyThis post details the redesign of Shopify's BFCM live map infrastructure using Apache Flink to address scalability issues and introduce new metrics. The previous system, Cricket (a Go-based stateful streaming service), was unable to handle the projected volume of checkout events. The new solution leverages Flink for filtering irrelevant checkout events and processing new metrics like product trends and unique shoppers. Flink was placed on the critical path, allowing Cricket to handle a reduced event volume for existing metrics. New metrics were computed by Flink and relayed through Cricket. The product trends metric involved calculating sales quantity percentage change over time using a specific formula and tumbling windows. The Flink jobs achieved 100% uptime during BFCM. A minor issue with Redis CPU load due to message fetching and the memory footprint of the new metric was resolved by dropping unnecessary Redis state.
Advertisement
Jan 2021Apache Beam for Search: Getting Started by Hacking Time - ShopifyThis post introduces Apache Beam as a unified batch and stream processing system for search relevance. It details the core challenge of 'time' in streaming systems, explaining event time vs. processing time and the role of watermarks. The post demonstrates configuring the timestamp at the source using Kafka's create time and introduces the concept of TimestampPolicy for custom event time extraction from data models like SearchQueryEvent. It highlights the importance of correctly handling time for accurate data processing in streaming pipelines.
Dec 2020How to Reliably Scale Your Data Platform for High Volumes - ShopifyThis post details Shopify's approach to scaling its data platform for high-volume events like BFCM. It outlines a tiered services taxonomy for prioritizing reliability, identifies key objectives and scaling levers for ingestion (Longboat) and query serving (Reportify) systems, and emphasizes the importance of load testing to validate mitigation strategies. The team focused on ensuring data freshness SLOs for ingestion and latency SLOs for query serving, and scaled clusters and tuned systems based on test results.
Nov 2020How to Build a Production Grade Workflow with SQL Modelling - ShopifyThis post details the creation of Seamster, a new SQL-based data pipeline workflow leveraging dbt and BigQuery. It introduces modular data model development with a directory-based structure for sources and models, implements Python-based unit testing with mock data and Great Expectations, and provides schema and DAG validation using temporary BigQuery datasets. The system aims to reduce development time and improve the consistency of business metrics by leveraging SQL for reporting tasks.
Aug 2020How to Track State with Type 2 Dimensional Models - ShopifyThis post details how to implement Type 2 dimensional models for tracking historical state in data warehouses, specifically for use cases like tracking user language preferences over time. It explains the limitations of Type 1 dimensions for analytical queries and presents three options for tracking state: modifying core application models, stitching together database snapshots, and adding database event logging. The post then provides detailed recipes for building Type 2 dimensions from event logs using PySpark and dbt, demonstrating how to transform event data into a Type 2 dimension with `valid_from`, `valid_to`, and `is_current` fields.
Nov 2018Running Apache Kafka on Kubernetes at Shopify - ShopifyThis post details Shopify's migration of its Apache Kafka infrastructure from on-premises data centers to Google Cloud Platform (GCP) and its subsequent deployment on Kubernetes. It outlines the three-step migration process, the rationale for choosing Kubernetes over VMs (specifically the use of StatefulSets and Custom Resources), best practices for containerizing Kafka (building own images, resource monitoring containers), and key configurations for cluster availability (Node Affinity, Taints, Inter-pod Affinity/Anti-Affinity, Persistent Volumes, Custom Resources for topic provisioning, rack-awareness, and Readiness Probes).