Adios to Busy-Waiting for Microsecond-scale Memory Disaggregation

How fast and efficiently page faults are handled determines the performance of paging-based memory disaggregation (MD) systems. Recent MD systems employ busy-waiting in page fault handling to avoid costly interrupt handling and context switching. Upon a page fault, they issue a remote fetch request and busy-wait for the completion of the request rather than yield their execution to other tasks. While these attempts succeed to cut the latency of MD systems to microseconds, they suffer from head-of-line (HOL) blocking that leads to high tail latency and causes RDMA network underutilization.

To address the problems, we reload the yield-based mechanism into the page fault handling and propose a new MD system, Adios. While yielding involves the switching overhead, Adios minimizes it by putting the page fault handler and the execution scheduler into a single address space. Then we use newly designed lightweight user-level threads, namely unithread. We also devise a dispatching algorithm that alleviates the imbalance in RDMA queue pairs, assuring lessened queueing delays and improved RDMA network utilization. Our evaluation demonstrates that Adios outperforms an existing state-of-the-art busy-waiting MD system, DiLOS, by up to 1.07-1.64× in throughput and 1.99-10.89× in P99.9 latency on real-world applications.