QLLMS: Quantization-Adaptive LLM Scheduling for Partially Informed Edge Serving Systems

The quantization technologies have enabled the practical deployment of large language models (LLMs) at the edge, however, current edge scheduling and quantization selection are separately designed. Furthermore, partially informed edge processing performance further exacerbates these challenges. To address these issues, we introduce QLLMS, a joint quantization-adaptive scheduling scheme for large language models tailored to partially informed edge serving systems. The primary objective of our approach is to reduce GPU rental costs by strategically orchestrating both quantization options and heterogeneous resources. The QLLMS algorithm first determines the available quantization set (AQS) to optimize the LLM inference performance within limited edge resources. To address the unpredictable nature of edge computing performance, we present a low-rank property-driven recovery approach, which can reconstruct complete AQS matrices using only partial samples. Subsequently, we devise a novel many-to-one matching algorithm that aims to strike a balance between efficient utilization of edge resources and optimal model inference performance, factoring in the available quantization options. We prove that QLLMS yields a stable matching without blocking pairs that could lead to inefficiencies. Experiment results show that QLLMS achieves a reduction of up to 22.36% in rental costs compared to state-of-the-art baselines in partially informed edge serving systems while simultaneously improving the task completion rate on edge servers.