Scaling Hierarchical Clustering

This post introduces techniques and tools for scaling hierarchical clustering. It details the computational complexity ($O(n^3)$ and $O(n^2 \log n)$) and memory usage issues associated with naive and optimized agglomerative approaches, particularly the large distance matrix. It then presents scaling techniques including sampling and approximation algorithms, and highlights tools like Fastcluster, Apache Spark, and GPU-accelerated solutions for handling large datasets.