SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning

Learning a stable and generalizable centralized value function (CVF) is a crucial but challenging task in multiagent reinforcement learning (MARL), as it has to deal with the issue that the joint action space increases exponentially with the number of agents in such scenarios. This article proposes an approach, named SMIX(<inline-formula> <tex-math notation="LaTeX">${\lambda }$ </tex-math></inline-formula>), that uses an OFF-policy training to achieve this by avoiding the greedy assumption commonly made in CVF learning. As importance sampling for such OFF-policy training is both computationally costly and numerically unstable, we proposed to use the <inline-formula> <tex-math notation="LaTeX">${\lambda }$ </tex-math></inline-formula>-return as a proxy to compute the temporal difference (TD) error. With this new loss function objective, we adopt a modified QMIX network structure as the base to train our model. By further connecting it with the <inline-formula> <tex-math notation="LaTeX">${Q(\lambda)}$ </tex-math></inline-formula> approach from a unified expectation correction viewpoint, we show that the proposed SMIX(<inline-formula> <tex-math notation="LaTeX">${\lambda }$ </tex-math></inline-formula>) is equivalent to <inline-formula> <tex-math notation="LaTeX">${Q(\lambda)}$ </tex-math></inline-formula> and hence shares its convergence properties, while without being suffered from the aforementioned curse of dimensionality problem inherent in MARL. Experiments on the StarCraft Multiagent Challenge (SMAC) benchmark demonstrate that our approach not only outperforms several state-of-the-art MARL methods by a large margin but also can be used as a general tool to improve the overall performance of other centralized training with decentralized execution (CTDE)-type algorithms by enhancing their CVFs.