Joint Policy Search for Multi-agent Collaboration with Imperfect Information

To learn good joint policies for multi-agent collaboration with imperfect information remains a fundamental challenge. While for two-player zero-sum games, coordinate-ascent approaches (optimizing one agent's policy at a time, e.g., selfplay [35, 20] ) work with guarantees, in multi-agent cooperative setting they often converge to sub-optimal Nash equilibrium. On the other hand, directly modeling joint policy changes in imperfect information game is nontrivial due to complicated interplay of policies (e.g., upstream updates affect downstream state reachability). In this paper, we show global changes of game values can be decomposed to policy changes localized at each information set, with a novel term named policychange density. Based on this, we propose Joint Policy Search (JPS) that iteratively improves joint policies of collaborative agents in imperfect information games, without re-evaluating the entire game. On multi-agent collaborative tabular games, JPS is proven to never worsen performance and can improve solutions provided by unilateral approaches (e.g, CFR [44]), outperforming algorithms designed for collaborative policy learning (e.g. BAD [16]). Furthermore, for real-world game with exponential states, JPS has an online form that naturally links with gradient updates. We test it to Contract Bridge, a 4-player imperfect-information game where a team of 2 collaborates to compete against the other. In its bidding phase, players bid in turn to find a good contract through a limited information channel. Based on a strong baseline agent that bids competitive bridge purely through domain-agnostic self-play, JPS improves collaboration of team players and outperforms WBridge5, a championship-winning software, by +0.63 IMPs (International Matching Points) per board over 1000 games, substantially better than previous SoTA (+0.41 IMPs/b against WBridge5) under Double-Dummy evaluation. Note that +0.1 IMPs/b is regarded as a nontrivial improvement in Computer Bridge. Part of the code is released in https://github.com/facebookresearch/jps . * Author contribution: Yuandong Tian proposed the theoretical framework, algorithm, proofs, did the majority of the code implementation and achieved SoTA performance. Qucheng Gong coded the logic of Bridge game, got initial results of A2C baselines, did ablation studies on the proposed method, designed simple games, provided insights in Contract Bridge, and built automatic UI tools to evaluate trained models against WBridge5. Tina Jiang performed experiments on baselines and ablations on simple games.