How We’re Solving Data Discovery Challenges at Shopify - Shopify

This post details the development of Artifact, Shopify's internal data discovery and management tool, to address challenges in data curation, governance, and accessibility. Artifact centralizes metadata from various data sources (tables, reports, dashboards, etc.) to improve data discovery, reduce duplication of effort, and enhance data governance. It provides a search and browse interface, displays data asset details including lineage powered by a graph database, and has a generic architecture designed for future integrations. The decision to build rather than buy was made due to the unique nature of Shopify's data stack and the need for a holistic view of data discovery processes. The metadata ingestion pipeline pulls data from various sources, with a future optimization planned for metadata APIs.