Tlaloc: A Generic Multipath Load Balancing for RoCE

As Remote Direct Memory Access (RDMA) is applied in larger data centers, load balancing is more necessary to reduce congestion hotspots and handle failures. However, the existing load balancing solutions are hard to be deployed in practical data centers with heterogenous RDMA NICs (RNICs) since they either work in Linux kernel or require to modify RNICs. In this paper, we propose Tlaloc, a general-purpose load balancing solution with high performance and low resource consumption. We overcome the information and operational limitations caused by the lack of transmission logic on the host and make it compatible with heterogeneous RNICs. To this end, we make the following three efforts: 1) collect accurate network information in a timely manner through message splitting and pace control; 2) dynamically maintain a high-quality set of connections to resist uncertainty; 3) establish a consistent-pace load scheduling framework to achieve elastic load allocation. We deploy Tlaloc on commercial RNIC drivers. Testbed experiments show that Tlaloc can improve network utilization by up to 53% and improve the speed of distributed training by up to 34% compared to ECMP. In large-scale simulations, Tlaloc improves FCT by up to 20% compared to others under real workloads.