Symphony: Enhancing RDMA Connection Scalability through Sender-Receiver Coordination

Remote Direct Memory Access (RDMA) has emerged as a cornerstone of datacenter networks (DCNs). Offloading the protocol stack to RDMA NIC (RNIC) makes data transmission depend on metadata caches, such as Queue Pair Context (QPC) and Work Queue Element (WQE). As DCNs expand, the number of Queue Pairs (QPs) grows, RNIC’s limited cache capacity becomes insufficient, triggering frequent cache misses and leading to serious connection scalability issues. Hardware-based solutions are prohibitively expensive. Existing software-based approaches mainly focus on QPC cache. Some attempt to optimize WQE cache, but only address Send WQEs (SWQEs). In this paper, to deal with unpredictable message arrival for optimizing Receive WQEs (RWQEs), we propose Symphony, a coordinated sender-receiver approach along with immediate data driven feedback that optimizes both SWQE and RWQE cache. We validate Symphony through NS3 simulation and a prototype implementation in RDMA driver, we also analyze its stability via the Laplace domain. The simulation results demonstrate that Symphony outperforms Weir in diverse scenarios, and testbed experiments indicate that Symphony can maintain 96Gbps with low CPU overhead. Compared to Weir, Symphony prominently reduces observable cache misses, ICM by approximately 20%, MTT/MPT by approximately 24% and RWQE by approximately 74%.