Observability-Aided Gpu Memory Oversubscription

Unified Virtual Memory (UVM) enables the oversubscription of GPUs' limited High Bandwidth Memory (HBM) capacity. Unfortunately, applications can significantly slow down when HBM is oversubscribed. We set out to make oversubscription practical without needing custom hardware modifications. The UVM driver, executing on the CPU, is responsible for eviction and prefetching but lacks observability into GPUs' accesses to HBM-resident memory. This fundamentally limits the driver's ability to make informed decisions. Toward this, we repurpose existing hardware access counters originally designed to track PCIe accesses to CPU's DRAM to aid page migration. Instead, we leverage the counters to provide (sampled) observability into GPU's accesses to HBM-resident pages. We then create ObservUVM, a novel software framework that enables easy exploration of observability-aided custom eviction and prefetching policies in the userspace, facilitating future research. We demonstrate that better-informed eviction and prefetching policies, enabled by the newfound observability, can significantly speed up GPU applications under memory oversubscription.