Improving GPU Multi-tenancy with Page Walk Stealing

GPU (Graphics Processing Unit) architecture has evolved to accelerate parts of a single application at a time. Consequently, several aspects of its architecture, particularly the virtual memory, have embraced a shared-mostly design. This implicitly assumes that a single application and, thus, one address space is resident in the GPU at a time. However, recent trends, e.g., deployment of GPUs in the cloud, necessitate efficient multi-tenancy. Multi-tenancy is needed for sharing the physical resources of a large server-class GPU across multiple concurrent tenants (applications) for resource consolidation while ensuring fairness among the tenants.

We first quantify how different components of GPU's virtual memory can impede multi-tenancy. We show that shared page walkers are a key bottleneck under multi-tenancy. We, therefore, propose dynamic page walk stealing that enables soft partitioning of the shared pool of walkers -reducing destructive interference between the tenants while also aggregating resources where possible. Over today's design, we improve throughput by 37%, and weighted IPC by 15%, on average, over 45 workloads.