How to Build a Production Grade Workflow with SQL Modelling - Shopify

This post details the creation of Seamster, a new SQL-based data pipeline workflow leveraging dbt and BigQuery. It introduces modular data model development with a directory-based structure for sources and models, implements Python-based unit testing with mock data and Great Expectations, and provides schema and DAG validation using temporary BigQuery datasets. The system aims to reduce development time and improve the consistency of business metrics by leveraging SQL for reporting tasks.