Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

In fully cooperative environments, agents aim to learn a joint policy to achieve a shared goal. However, existing Multi-Agent Reinforcement Learning (MARL) approaches struggle when scaling to complex coordination tasks. However, as the complexity of joint tasks increases and the policy space expands, agents face significant challenges in achieving the convergence of optimal policies. The limited observational capabilities of agents, coupled with time-varying interaction weights among neighboring agents, lead to challenges in maintaining stable policy evaluations. To address these challenges, we propose GDE, a MARL framework that combines Graph-based value Decomposition with staged Evolutionary policy optimization. To enhance the efficiency of policy exploration and convergence, we use Evolutionary Algorithms (EAs) with diverse in-population characteristics to conduct gradient-free random search. We employ Graph Neural Networks (GNNs) to extend agents' receptive fields, improving information propagation across neighbors and enhancing coordination in dynamic environments without requiring state consensus. Furthermore, the permutation invariance of topological graphs allows GNNs to maintain stable convergence when processing dynamic data. The formation of multiple agent teams enhances GNNs' ability to capture complex coordination dynamics within the multi-agent system. Our method enables staged optimization of agent policies through evolutionary mechanisms while continuously updating joint policies based on graph relationships. Experiments conducted on micro-management in StarCraft II, robot cooperation in MAMuJoCo, and autonomous driving in SUMO demonstrate the superior performance of GDE, validating the effectiveness and necessity of each proposed module. Our code is available: https://github.com/MercyM/GDE.