TianheEngine: Hierarchy-aware Adaptive Partitioning System for Trillion-scale Graph Processing

Graph partitioning is essential for effectively managing multi-trillion-edge graphs in distributed computing systems, particularly those spanning hierarchical architectures with thousands of computing nodes. Traditional partitioning strategies neglect hierarchical communication variances across modern high-performance computing (HPC) systems, leading to prohibitive cross overhead when processing trillion-scale graphs. We propose TianheEngine, a hierarchy-aware adaptive partitioning system that takes advantage of the communication hierarchy of the underlying distributed computing system and the sparsity characteristics of the input graphs to improve communication efficiency. We evaluated TianheEngine on fundamental graph operations using both synthetic and real-world datasets. Our extensive experiments use up to 79,024 computing nodes and over 1.2 million processor cores. Experimental results show that TianheEngine is superior to state-of-the-art graph partitioning methods and parallel graph systems and outperforms top-ranked systems on the latest Graph 500 list.