DEMUS: Large Multimodal Model Serving at the Edge with Diffusion-Based Scheduling

Deploying large multimodal models (LMMs) on the edge is a promising solution for latency-sensitive applications, while also addressing the high bandwidth costs and privacy concerns associated with cloud-based serving. LMM inference involves a compute-intensive context stage and a memory-intensive generation stage. However, existing LMM serving systems typically colocate both stages within a single instance, which precludes independent optimization and leads to inefficient resource utilization and increased latency, especially in resource-constrained edge settings. In this paper, we present DEMUS, a Decoupled edge serving framework for large Multimodal models with diffusion-based Scheduling. DEMUS decouples the two stages to optimize them independently without interference, and schedules context and generation tasks to corresponding instances. It combines diffusion-based generation with model predictive control to effectively schedule tasks across instances on heterogeneous nodes, aiming to maximize overall system throughput under latency constraints. We implement and evaluate DEMUS on a heterogeneous cluster with 16 edge server nodes and 45 GPUs. Extensive experiments shows that DEMUS achieves 2.1 3.7× higher throughput while meeting latency SLOs, and generalizes well in dynamic environments and unseen settings.