Capturing Every Change From Shopify’s Sharded Monolith - Shopify

This post details the limitations of Shopify's existing Longboat system for data extraction, including its inability to capture hard deletes, reliance on the `updated_at` field, missing intermediate row states, consistency issues, and missed records. It introduces a new approach for the Merchant Analytics Platform that uses Kafka for change-event capture and streaming transformations, enabling near real-time data ingestion and lower latency query responses, addressing the need for fresher and more comprehensive merchant analytics.