Large-scale graph partitioning with Apache Giraph

This post introduces a novel approach to large-scale graph partitioning using Apache Giraph to optimize data distribution in sharded systems. It addresses the challenge of high latency in queries that require fetching data across multiple machines by co-locating related data (e.g., friends of a person) within the same partition. The post details a heuristic algorithm inspired by balanced label propagation, adapted for Giraph's distributed model, which iteratively swaps vertices to maximize local edges while maintaining partition balance. It demonstrates significant improvements in iteration time for graph processing algorithms like PageRank and a 2x improvement in bandwidth utilization for a specific service by improving data locality.