TechPrufer
Get started
Hex
Hex logoHex · Feature trail

Scaling Hierarchical Clustering

2 posts · Oct 2023

What this trail covers

Hex has evolved its data science capabilities by detailing techniques and tools for scaling hierarchical clustering. This includes exploring the computational and memory challenges of traditional methods, and introducing solutions like sampling, approximation algorithms, and leveraging specialized libraries and distributed computing frameworks such as Fastcluster, Apache Spark, and GPU-accelerated approaches. The post aims to provide practical strategies for applying hierarchical clustering to large datasets.

All Hex trails
  1. Comparing DBSCAN, k-means, and Hierarchical Clustering: When and Why To Choose Density-Based MethodsThis post details the comparison of DBSCAN, k-means, and hierarchical clustering algorithms, focusing on their methodologies, strengths, and limitations. It provides code examples for k-means and hierarchical clustering using the iris dataset and discusses suitable scenarios for each. The post also introduces DBSCAN's core concepts (core points, border points, noise points) and its parameters (epsilon and MinPts), highlighting its ability to discover clusters of arbitrary shape and handle noise, making it suitable for complex data distributions.
  2. Scaling Hierarchical ClusteringThis post introduces techniques and tools for scaling hierarchical clustering. It details the computational complexity ($O(n^3)$ and $O(n^2 \log n)$) and memory usage issues associated with naive and optimized agglomerative approaches, particularly the large distance matrix. It then presents scaling techniques including sampling and approximation algorithms, and highlights tools like Fastcluster, Apache Spark, and GPU-accelerated solutions for handling large datasets.