Frugal: Efficient and Economic Embedding Model Training with Commodity GPUs

Embedding models show superiority in learning representations of massive ID-type features in sparse learning scenarios such as recommendation systems (e.g., user/item IDs) and graph learning (e.g., node/edge IDs). Commodity GPUs are highly favored for their cost-efficient computing power, which is ideally suited for the low computing demand of memory-intensive embedding models. However, directly running embedding model training on commodity GPUs yields poor performance because of their deficient communication resources (including low communication bandwidth and no PCIe P2P support).

This paper presents Frugal, an embedding model training system tailored for commodity GPUs. Based on the observation that the communication between commodity GPUs must be bounced on host memory (due to no PCIe P2P support), the key idea of Frugal is proactively flushing, where each GPU proactively flushes its own parameters that other GPUs will access into host memory, thereby decoupling half of the communication overhead to non-critical paths. To alleviate the communication contention of proactively flushing on foreground training processes, Frugal assigns priorities to each flush operation, and prioritizes flushing parameters that GPUs will access while deferring others. Further, Frugal tailors a two-level priority queue to ensure high scalability for operations involving priorities. Frugal has been applied to train embedding models including recommendation models and graph embedding. Experiments indicate that Frugal