PC4: Precision Collective Communication Congestion Control for AI Cluster

The evolution of AI has driven the trend towards utilizing larger models with increasing parameter sizes, necessitating deployment in datacenters comprising tens of thousands of GPUs. However, as cluster sizes expand, the overall computational capacity of the system fails to scale linearly with the number of compute nodes. This non-linear scaling law is primarily due to decreased GPU utilization caused by network congestion, which significantly impacts model training speed. In this paper, we analyze the network topology and traffic characteristics of AI clusters and propose PC4, a precision collective communication congestion control algorithm tailored for AI clusters. PC4 optimizes congestion control based on collective communication operations (e.g., all-reduce, all-to-all) inherent in collective communication libraries (e.g., MPI, NCCL) used in distributed training. By leveraging information from these collective communication operations, we compute a base rate that enables rapid system convergence. Furthermore, PC4 employs a datacenter time synchronization mechanism, utilizing one-way delay as congestion signal. This approach allows for more precise control of link queue lengths and queuing time, thereby efficiently managing congestion. In our all-to-all evaluation, PC4 achieves a 72% reduction in tail FCT compared to DCQCN.