Forest: Access-aware GPU UVM Management

With GPU unified virtual memory (UVM), CPU and GPU can share a flat virtual address space. UVM enables the GPUs to utilize the larger CPU system memory as an expanded memory space. However, UVM’s on-demand page migration is accompanied by expensive page fault handling overhead. To mitigate such overhead, tree-based neighboring prefetcher (TBNp) has been used by GPUs. TBNp effectively reduces page faults by exploiting locality at multiple levels. However, we observe its access-pattern oblivious design leads to excessive page thrashing and unnecessary migrations. In this paper, we tackle the inefficiencies with a novel access-aware UVM management, Forest. Forest uses a software-hardware codesign to configure the optimal tree prefetchers at runtime based on each data object’s access patterns. With the heterogeneous tree-based prefetching, Forest provides 1.86 × and 1.39 × speedups over the baseline TBNp and state-of-the-art optimization solutions, respectively. Forest also shows a 1.51 × speedup for real-world deep learning models, including CNNs and Transformers.