Why We Use Separate Tech Stacks for Personalization and Experimentation

This post details Spotify's technical and practical reasons for separating personalization and experimentation stacks. It explains that personalization systems, including contextual bandits, reside in the ML stack due to strict infrastructure and performance requirements for model training and low-latency serving. Experimentation systems, like the Confidence platform, are used to evaluate these personalization systems through A/B tests. The post argues against using non-contextual multi-armed bandits for experimentation due to their one-dimensional focus on single metrics and the need for patience with business metrics, advocating for robust A/B testing instead. It emphasizes treating personalization systems as treatment variants within A/B tests and leveraging tight integration between the ML and experimentation platforms.