Recursive Embedding and Clustering

This post introduces a new capability for recursive clustering and embedding. It details the use of UMAP for dimensionality reduction and HDBSCAN for robust clustering, followed by a recursive application to zoom into sub-clusters. To address the complexity and lack of explainability in these stacked algorithms, a supervised learning model (XGBoost) is trained on the cluster assignments, with SHAP values used to explain the model's predictions and thus the underlying clustering logic. This enables a deeper understanding of user behavior and provides a production-ready model for classification.