Shepherd: How Stripe adapted Chronon to scale ML feature development

This post details how Stripe adapted Chronon, an open-source data lineage and feature definition framework, to build Shepherd, a platform for scaling ML feature development. It highlights the technical challenges of managing feature definitions, data lineage, and integration with existing infrastructure at Stripe's scale. Key technical contributions include the adaptation of Chronon's schema and lineage tracking capabilities to support ML feature development, the implementation of a robust feature definition language, and the integration of Shepherd with Stripe's data processing and model training pipelines.