Lune

NeurIPS2022Top-tier venue

Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learning

Dilip Arumugam, Benjamin Van Roy

Abstract

The quintessential model-based reinforcement-learning agent iteratively refines its estimates or prior beliefs about the true underlying model of the environment. Recent empirical successes in model-based reinforcement learning with function approximation, however, eschew the true model in favor of a surrogate that, while ignoring various facets of the environment, still facilitates effective planning over behaviors. Recently formalized as the value equivalence principle, this algorithmic technique is perhaps unavoidable as real-world reinforcement learning demands consideration of a simple, computationally-bounded agent interacting with an overwhelmingly complex environment, whose underlying dynamics likely exceed the agent's capacity for representation. In this work, we consider the scenario where agent limitations may entirely preclude identifying an exactly value-equivalent model, immediately giving rise to a trade-off between identifying a model that is simple enough to learn while only incurring bounded sub-optimality. To address this problem, we introduce an algorithm that, using rate-distortion theory, iteratively computes an approximately-value-equivalent, lossy compression of the environment which an agent may feasibly target in lieu of the true model. We prove an information-theoretic, Bayesian regret bound for our algorithm that holds for any finite-horizon, episodic sequential decision-making problem. Crucially, our regret bound can be expressed in one of two possible forms, providing a performance guarantee for finding either the simplest model that achieves a desired sub-optimality gap or, alternatively, the best model given a limit on agent capacity.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective cites Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning
  • FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs cites Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning
  • Improved Bayesian Regret Bounds for Thompson Sampling in Reinforcement Learning cites Model-Based Reinforcement Learning with Value-Targeted Regression
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Toward Efficient Exploration by Large Language Model AgentsDilip Arumugam, Thomas L. GriffithsICLR 2026 · 17 citations
  2. Distributional Model Equivalence for Risk-Sensitive Reinforcement LearningTyler Kastner, Murat A. Erdogdu, Amir-massoud FarahmandNeurIPS 2023 · 9 citations
  3. Improved Bayesian Regret Bounds for Thompson Sampling in Reinforcement LearningAhmadreza Moradipari, Mohammad Pedramfar, Modjtaba Shokrian Zini, Vaneet AggarwalNeurIPS 2023 · 8 citations
  4. Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One ObjectiveRaj Ghugare, Homanga Bharadhwaj, Benjamin Eysenbach, Sergey Levine et al.ICLR 2023 · 2 citations
  5. On Bits and Bandits: Quantifying the Regret-Information Trade-offItai Shufaro, Nadav Merlis, Nir Weinberger, Shie MannorICLR 2025

Builds on18

  1. Model-Based Reinforcement Learning with Value-Targeted RegressionAlex Ayoub, Zeyu Jia, Csaba Szepesvári, Mengdi Wang et al.ICML 2020 · 324 citations
  2. VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-LearningLuisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze et al.ICLR 2020 · 315 citations
  3. FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPsAlekh Agarwal, Sham M. Kakade, Akshay Krishnamurthy, Wen SunNeurIPS 2020 · 271 citations
  4. Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement LearningDipendra Misra, Mikael Henaff, Akshay Krishnamurthy, John LangfordICML 2020 · 158 citations
  5. Epistemic Neural NetworksIan Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla et al.NeurIPS 2023 · 142 citations

Related papers

  1. Approximate Value EquivalenceChristopher Grimm, André Barreto, Satinder SinghNeurIPS 2022 · 7 citations
  2. The Value Equivalence Principle for Model-Based Reinforcement LearningChristopher Grimm, André Barreto, Satinder Singh, David SilverNeurIPS 2020 · 129 citations
  3. Deciding What to Learn: A Rate-Distortion ApproachDilip Arumugam, Benjamin Van RoyICML 2021 · 29 citations
  4. Proper Value EquivalenceChristopher Grimm, André Barreto, Gregory Farquhar, David Silver et al.NeurIPS 2021 · 49 citations
  5. Reinforcement Learning with Simple Sequence PriorsTankred Saanum, Noémi Élteto, Peter Dayan, Marcel Binz et al.NeurIPS 2023 · 18 citations
Dusk over the sea between two cliffs drawn in fine vertical lines