Beyond Rule-Based Agents: Active Markov Games for Realistic Multi-Agent Interaction in Autonomous Driving

Current research in autonomous driving heavily relies on large-scale driving datasets for model fitting or trial-and-error learning strategies in simulation environments. However, these approaches suffer from limited behavioral diversity and fail to cover complex edge-case interactions. To address these limitations, we model the driving environment as an Active Markov Game (AMG) and introduce a multi-agent co-evolutionary training framework for more realistic interactive learning. The AMG formulation extends traditional Markov games by explicitly making state transitions and rewards dependent on the evolving strategies of the agents, thus capturing the interactive dynamics and strategic coupling between the ego vehicle and surrounding agents. Building on this, our multi-agent co-evolutionary training mechanism jointly optimizes the ego vehicle's policy and a diverse pool of opponent strategies, allowing all agents to adapt to each other's behaviors during training. This game-theoretic approach produces a robust ego agent capable of handling diverse, non-stationary driving strategies, overcoming the "non-responsive opponent" limitation found in prior methods. In CARLA simulations of unsignalized intersections and long-tail scenarios, our method performs exceptionally well, achieving near-perfect success rates (98%) with minimal collisions (2%), and significantly outperforming state-of-the-art baselines such as PPO, DDPG, and IPPO in terms of generalization, safety margins, and control smoothness. These results demonstrate that our approach substantially enhances the robustness, safety, and strategic adaptability of autonomous driving in complex multi-agent environments.