Graph Diffusion for Robust Multi-Agent Coordination

Offline multi-agent reinforcement learning (MARL) struggles to estimate out-of-distribution states or actions due to the absence of real-time interactions with the environment. Although diffusion models have shown promising potential in addressing these challenges, they primarily apply independent diffusion to the historical trajectories of individual agents, which overlooks the crucial dynamics in multi-agent coordination and limits the policy robustness in dynamic environments. In this paper, we propose MCGD, a novel Multi-agent Coordination framework based on Graph Diffusion models to improve the effectiveness and robustness of collaborative policies. Specifically, we construct a sparse coordination graph with continuous node attributes and discrete edge attributes to identify the underlying multi-agent dynamics effectively. We then derive the transition probabilities between edge categories and present adaptive categorical diffusion to model the structure diversity of inter-agent coordination. According to the coordination structure, we define the neighbor-dependent forward noise and design anisotropic diffusion to increase the action diversity of each agent. Extensive experiments across various multi-agent environments demonstrate that MCGD significantly outperforms existing state-of-the-art baselines in coordination performance and exhibits superior robustness to dynamic environmental changes.