Policy improvement by planning with Gumbel
Ivo Danihelka, Arthur Guez, Julian Schrittwieser, David Silver
Abstract
AlphaZero is a powerful reinforcement learning algorithm based on approximate policy iteration and tree search. However, AlphaZero can fail to improve its policy network, if not visiting all actions at the root of a search tree. To address this issue, we propose a policy improvement algorithm based on sampling actions without replacement. Furthermore, we use the idea of policy improvement to replace the more heuristic mechanisms by which AlphaZero selects and uses actions, both at root nodes and at non-root nodes. Our new algorithms, Gumbel AlphaZero and Gumbel MuZero, respectively without and with model-learning, match the state of the art on Go, chess, and Atari, and significantly improve prior performance when planning with few simulations.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
30 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Learning and Planning in Complex Action Spaces cites Monte-Carlo Tree Search as Regularized Policy Optimization
- EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data cites Learning and Planning in Complex Action Spaces
- EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data cites Online and Offline Reinforcement Learning by Planning with a Learned Model
- Efficient Multi-agent Reinforcement Learning by Planning cites Learning and Planning in Complex Action Spaces
- Opponent Modeling with In-context Search cites Learning and Planning in Complex Action Spaces
- SPO: Sequential Monte Carlo Policy Optimisation cites Learning and Planning in Complex Action Spaces
- SPO: Sequential Monte Carlo Policy Optimisation cites Monte-Carlo Tree Search as Regularized Policy Optimization
- floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL cites Learning and Planning in Complex Action Spaces
- Online and Offline Reinforcement Learning by Planning with a Learned Model cites Learning and Planning in Complex Action Spaces
Cited by top-tier papers30
- EfficientZero V2: Mastering Discrete and Continuous Control with Limited DataShengjie Wang, Shaohuai Liu, Weirui Ye, Jiacheng You et al.ICML 2024 · 36 citations
- floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLBhavya Agrawalla, Michal Nauman, Khush Agrawal, Aviral KumarICLR 2026 · 22 citations
- Efficient Multi-agent Reinforcement Learning by PlanningQihan Liu, Jianing Ye, Xiaoteng Ma, Jun Yang et al.ICLR 2024 · 18 citations
- Opponent Modeling with In-context SearchYuheng Jing, Bingyun Liu, Kai Li, Yifan Zang et al.NeurIPS 2024 · 8 citations
- SPO: Sequential Monte Carlo Policy OptimisationMatthew Macfarlane, Edan Toledo, Donal Byrne, Paul Duckworth et al.NeurIPS 2024 · 8 citations
Builds on7
- Critic Regularized RegressionZiyu Wang, Alexander Novikov, Konrad Zolna, Josh Merel et al.NeurIPS 2020 · 406 citations
- Online and Offline Reinforcement Learning by Planning with a Learned ModelJulian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain et al.NeurIPS 2021 · 149 citations
- Learning and Planning in Complex Action SpacesThomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain et al.ICML 2021 · 99 citations
- Monte-Carlo Tree Search as Regularized Policy OptimizationJean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert et al.ICML 2020 · 84 citations
- Estimating Gradients for Discrete Random Variables by Sampling without ReplacementWouter Kool, Herke van Hoof, Max WellingICLR 2020 · 59 citations
Related papers
- Multiagent Gumbel MuZero: Efficient Planning in Combinatorial Action SpacesXiaotian Hao, Jianye Hao, Chenjun Xiao, Kai Li et al.AAAI 2024 · 5 citations
- Planning in Stochastic Environments with a Learned ModelIoannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K. Hubert et al.ICLR 2022 · 79 citations
- On the role of planning in model-based deep reinforcement learningJessica B. Hamrick, Abram L. Friesen, Feryal M. P. Behbahani, Arthur Guez et al.ICLR 2021 · 77 citations
- OptionZero: Planning with Learned OptionsPo-Wei Huang, Pei-Chiun Peng, Hung Guei, Ti-Rong WuICLR 2025
- Accelerating Monte Carlo Tree Search with Probability Tree State AbstractionYangqing Fu, Ming Sun, Buqing Nie, Yue GaoNeurIPS 2023 · 5 citations
