Lune

ICML2024Top-tier venue

A Distributional Analogue to the Successor Representation

Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Yunhao Tang, André Barreto, Will Dabney, Marc G. Bellemare, Mark Rowland

Abstract

This paper contributes a new approach for distributional reinforcement learning which elucidates a clean separation of transition structure and reward in the learning process. Analogous to how the successor representation (SR) describes the expected consequences of behaving according to a given policy, our distributional successor measure (SM) describes the distributional consequences of this behaviour. We formulate the distributional SM as a distribution over distributions and provide theory connecting it with distributional and model-based reinforcement learning. Moreover, we propose an algorithm that learns the distributional SM from data by minimizing a two-level maximum mean discrepancy. Key to our method are a number of algorithmic techniques that are independently valuable for learning generative models of state. As an illustration of the usefulness of the distributional SM, we show that it enables zero-shot risk-sensitive policy evaluation in a way that was not previously possible.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Reinforcement Learning from Passive Data via Latent Intentions cites Contrastive Learning as Goal-Conditioned Reinforcement Learning
  • Reinforcement Learning from Passive Data via Latent Intentions cites Learning One Representation to Optimize All Rewards
  • A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximation cites Foundations of Multivariate Distributional Reinforcement Learning
  • Reward-Aware Proto-Representations in Reinforcement Learning cites Count-Based Exploration with the Successor Representation
  • Reward-Aware Proto-Representations in Reinforcement Learning cites Learning One Representation to Optimize All Rewards
  • Towards Robust Zero-Shot Reinforcement Learning cites Learning One Representation to Optimize All Rewards
  • Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning cites Foundations of Multivariate Distributional Reinforcement Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. Foundations of Multivariate Distributional Reinforcement LearningHarley Wiltzer, Jesse Farebrother, Arthur Gretton, Mark RowlandNeurIPS 2024 · 21 citations
  2. Towards Robust Zero-Shot Reinforcement LearningKexin Zheng, Lauriane Teyssier, Yinan Zheng, Yu Luo et al.NeurIPS 2025 · 7 citations
  3. A Finite Sample Analysis of Distributional TD Learning with Linear Function ApproximationYang Peng, Kaicheng Jin, Liangyu Zhang, Zhihua ZhangNeurIPS 2025 · 6 citations
  4. Reward-Aware Proto-Representations in Reinforcement LearningHon Tik Tse, Siddarth Chandrasekar, Marlos C. MachadoNeurIPS 2025 · 6 citations
  5. Convergence Theorems for Entropy-Regularized and Distributional Reinforcement LearningYash Jhaveri, Harley Wiltzer, Patrick Shafto, Marc G. Bellemare et al.NeurIPS 2025 · 3 citations

Builds on19

  1. Contrastive Learning as Goal-Conditioned Reinforcement LearningBenjamin Eysenbach, Tianjun Zhang, Sergey Levine, Ruslan SalakhutdinovNeurIPS 2022 · 331 citations
  2. Count-Based Exploration with the Successor RepresentationMarlos C. Machado, Marc G. Bellemare, Michael BowlingAAAI 2020 · 206 citations
  3. Learning One Representation to Optimize All RewardsAhmed Touati, Yann OllivierNeurIPS 2021 · 140 citations
  4. Einops: Clear and Reliable Tensor Manipulations with Einstein-like NotationAlex RogozhnikovICLR 2022 · 124 citations
  5. Reinforcement Learning from Passive Data via Latent IntentionsDibya Ghosh, Chethan Anand Bhateja, Sergey LevineICML 2023 · 69 citations

Related papers

  1. Distributional Model Equivalence for Risk-Sensitive Reinforcement LearningTyler Kastner, Murat A. Erdogdu, Amir-massoud FarahmandNeurIPS 2023 · 9 citations
  2. A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor RepresentationScott Fujimoto, David Meger, Doina PrecupICML 2021 · 17 citations
  3. Distributional Successor Features Enable Zero-Shot Policy OptimizationChuning Zhu, Xinqi Wang, Tyler Han, Simon S. Du et al.NeurIPS 2024 · 11 citations
  4. Provable Risk-Sensitive Distributional Reinforcement Learning with General Function ApproximationYu Chen, Xiangcheng Zhang, Siwei Wang, Longbo HuangICML 2024 · 3 citations
  5. Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative ModelMark Rowland, Kevin Kevin Li, Rémi Munos, Clare Lyle et al.NeurIPS 2024 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines