How to Build a Production Grade Workflow with SQL Modelling - Shopify

What this post added

This post details the creation of Seamster, a new data pipeline workflow built on dbt and Google BigQuery, to address the limitations of the previous PySpark-based system (Starscream) for common reporting tasks. Key contributions include: modularizing data model development with a new source and model structure, implementing a Python-based unit testing framework with mock data and Great Expectations, and establishing CI pipelines for schema and DAG validation using temporary BigQuery datasets. The system aims to improve development time and thinking by favoring SQL for data modeling.