The Winding Road to Better Machine Learning Infrastructure Through Tensorflow Extended and Kubeflow

This post details Spotify's journey in building ML infrastructure, moving from early custom Scala-based tooling to standardizing on TensorFlow Extended (TFX) and Kubeflow Pipelines (KFP). It outlines three iterations: the first focused on connectors for existing Scala data tooling, the second on standardizing on TFX for its Python-centric ML community and robust components, and the third on adopting Kubeflow Pipelines for orchestration, component sharing, and experiment tracking. The post also highlights the challenges and solutions in integrating TFX and KFP, and the operational aspects of managing Kubeflow clusters on GCP, including open-sourcing a Terraform module for GKE cluster setup.