Capturing Every Change From Shopify’s Sharded Monolith - Shopify

What this post added

This post details the limitations of Shopify's existing batch-query-based data extraction service, Longboat, including its inability to capture hard deletes, reliance on the `updated_at` field, missing intermediate row states, consistency issues, and missed records. It also explains why simply increasing Longboat's query frequency is not feasible due to performance impacts on source datastores. The post sets the stage for a new unified data extraction pipeline that leverages Change Data Capture (CDC) to address these challenges and improve data freshness and accuracy.