Heterogeneous Mean-field Reinforcement Learning for Age-minimal GPU Batching

With the proliferation of real-time intelligent applications, maintaining information freshness, quantified by the Age of Information (AoI), has become a critical mission for edge intelligence systems. Batch processing has become an indispensable component in the efficient scheduling of computational resources. In this work, we study the first age-minimal GPU batching problem in a multi-agent environment, driven by the coupling between users’ data generation decisions and the server’s batching policy. To tackle the scalability challenge of the curse of dimensionality, we leverage heterogeneous mean-field reinforcement learning (HMFRL). We formulate the system as a heterogeneous multi-agent Semi-Markov Decision Process (SMDP) and design a scalable, asynchronous Proximal Policy Optimization (PPO) algorithm, termed HMF-PPO. This framework approximates the complex N-agent interactions with the simplified dynamics between a representative agent and a "mean field" that characterizes the collective behavior, thereby effectively mitigating non-stationarity and ensuring scalability. Theoretical analysis reveals that the mean-field approximation error converges to zero at the rate of ${\mathcal{O}}(1/\sqrt N )$. Experimental results demonstrate that HMF-PPO can achieve up to 47% AoI reduction compared to no-batching scheduling. It also outperforms leading baselines in terms of convergence stability and scalability, and support incremental online learning for dynamic real-world edge environments.