How We Built Infrastructure to Run User Forecasts at Spotify

This post details the development of a new infrastructure for running user forecasts at Spotify. The system enables on-demand and automated weekly forecasts with advanced hyperparameter tuning, significantly reducing model training times. It incorporates distinct workstreams for mature markets (time series models), new markets (proxy-based models), and custom/specialty markets requiring human input. The infrastructure leverages Dataflow for parallel processing and Dockerized Python jobs on Kubernetes. Key components for mature market forecasts include a Parameter-Space Creator, Data Labeler, Hyperparameter Tuning, Model Selector, Model Applier, QC & Manual Intervention tools, and a Publish & Visualization module. Learnings highlight the importance of balancing quality control with fast iteration, smart scaling choices, and system simplicity.