Recursive Clustering and Explainability

Spotify has developed a novel approach to clustering diverse datasets by combining dimensionality reduction (UMAP), hierarchical density-based clustering (HDBSCAN), and recursive application of these techniques to identify sub-clusters. This method is enhanced with a supervised machine learning layer (XGBoost with SHAP values) to provide explainability, allowing data scientists and user researchers to understand the drivers behind cluster formation and iterate faster on user insights. The system aims to make large, complex datasets manageable, discover nuanced user segments, and provide a validated explainability layer for findings.