RecFlow: Unlocking GPU Efficiency for DLRM Inference via Fine-Grained Parallelism and Incremental Batching

Industrial Deep Learning Recommendation Models (DLRMs) comprise memory-intensive embedding operations and compute-intensive DNN layers, often resulting in suboptimal GPU resource utilization under high-throughput inference workloads. However, the memory demands of DNN layers vary considerably across different workloads and execution phases, often leading to unpredictable interference, which makes it challenging to efficiently co-execute embedding and DNN operators.This paper presents RecFlow, a high-performance DLRM serving framework that leverages intra-SM parallelism to co-run embedding and DNN computations through fine-grained resource coordination. RecFlow profiles the workload characteristics of each DNN phase and applies adaptive parallel strategies to sustain high memory bandwidth utilization while minimizing interoperator interference. To further enable parallelism between the structurally dependent embedding and top-DNN stages, RecFlow introduces an incremental batching mechanism that overlaps their execution using newly arrived requests, thereby enabling inter-batch parallelism without incurring additional queuing latency. Extensive evaluations on real-world production workloads demonstrate that RecFlow improves serving throughput by up to 1.13 × over state-of-the-art DLRM inference systems while reducing latency in high-throughput serving scenarios.