Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods
Yi Wan, Ali Rahimi-Kalahroudi, Janarthanan Rajendran, Ida Momennejad, Sarath Chandar, Harm van Seijen
Abstract
In recent years, a growing number of deep model-based reinforcement learning (RL) methods have been introduced. The interest in deep model-based RL is not surprising, given its many potential benefits, such as higher sample efficiency and the potential for fast adaption to changes in the environment. However, we demonstrate, using an improved version of the recently introduced Local Change Adaptation (LoCA) setup, that well-known model-based methods such as PlaNet and DreamerV2 perform poorly in their ability to adapt to local environmental changes. Combined with prior work that made a similar observation about the other popular model-based method, MuZero, a trend appears to emerge, suggesting that current deep model-based methods have serious limitations. We dive deeper into the causes of this poor performance, by identifying elements that hurt adaptive behavior and linking these to underlying techniques frequently used in deep model-based RL. We empirically validate these insights in the case of linear function approximation by demonstrating that a modified version of linear Dyna achieves effective adaptation to local changes. Furthermore, we provide detailed insights into the challenges of building an adaptive nonlinear model-based method, by experimenting with a nonlinear version of Dyna.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Multi-Step Generalized Policy Improvement by Leveraging Approximate Models cites Mastering Atari with Discrete World Models
- Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning cites Same State, Different Task: Continual Reinforcement Learning without Interference
- Curious Replay for Model-based Adaptation cites Mastering Atari with Discrete World Models
- Curious Replay for Model-based Adaptation cites Dream to Control: Learning Behaviors by Latent Imagination
- Curious Replay for Model-based Adaptation cites The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning
- The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning cites Dream to Control: Learning Behaviors by Latent Imagination
Cited by top-tier papers5
- Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalIda Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma et al.NeurIPS 2023 · 114 citations
- Curious Replay for Model-based AdaptationIsaac Kauvar, Chris Doyle, Linqi Zhou, Nick HaberICML 2023 · 18 citations
- Multi-Step Generalized Policy Improvement by Leveraging Approximate ModelsLucas Nunes Alegre, Ana L. C. Bazzan, Ann Nowé, Bruno C. da SilvaNeurIPS 2023 · 7 citations
- Laplacian Representations for Decision-Time PlanningDikshant Shehmar, Matthew Schlegel, Matthew Taylor, Marlos C. MachadoICML 2026 · 1 citation
- Principled Fast and Meta Knowledge Learners for Continual Reinforcement LearningKe Sun, Hongming Zhang, Jun Jin, Chao Gao et al.ICLR 2026 · 1 citation
Builds on5
- Dream to Control: Learning Behaviors by Latent ImaginationDanijar Hafner, Timothy P. Lillicrap, Jimmy Ba, Mohammad NorouziICLR 2020 · 1,852 citations
- Mastering Atari with Discrete World ModelsDanijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, Jimmy BaICLR 2021 · 1,170 citations
- Same State, Different Task: Continual Reinforcement Learning without InterferenceSamuel Kessler, Jack Parker-Holder, Philip J. Ball, Stefan Zohren et al.AAAI 2022 · 57 citations
- Reset-Free Lifelong Learning with Skill-Space PlanningKevin Lu, Aditya Grover, Pieter Abbeel, Igor MordatchICLR 2021 · 42 citations
- The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement LearningHarm van Seijen, Hadi Nekoei, Evan Racah, Sarath ChandarNeurIPS 2020 · 16 citations
Related papers
- On the role of planning in model-based deep reinforcement learningJessica B. Hamrick, Abram L. Friesen, Feryal M. P. Behbahani, Arthur Guez et al.ICLR 2021 · 77 citations
- Trust the Model Where It Trusts Itself - Model-Based Actor-Critic with Uncertainty-Aware Rollout AdaptionBernd Frauenknecht, Artur Eisele, Devdutt Subhasish, Friedrich Solowjow et al.ICML 2024 · 14 citations
- Procedural generalization by planning with self-supervised world modelsAnkesh Anand, Jacob C. Walker, Yazhe Li, Eszter Vértes et al.ICLR 2022 · 34 citations
- SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement LearningZhongjian Qiao, Jiafei Lyu, Kechen Jiao, Qi Liu et al.AAAI 2025 · 12 citations
- DyMoDreamer: World Modeling with Dynamic ModulationBoxuan Zhang, Runqing Wang, Wei Xiao, Weipu Zhang et al.NeurIPS 2025 · 2 citations
