Lune

ICML2022Top-tier venue

Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning

Philippe Hansen-Estruch, Amy Zhang, Ashvin Nair, Patrick Yin, Sergey Levine

Abstract

Building generalizable goal-conditioned agents from rich observations is a key to reinforcement learning (RL) solving real world problems. Traditionally in goal-conditioned RL, an agent is provided with the exact goal they intend to reach. However, it is often not realistic to know the configuration of the goal before performing a task. A more scalable framework would allow us to provide the agent with an example of an analogous task, and have the agent then infer what the goal should be for its current state. We propose a new form of state abstraction called goal-conditioned bisimulation that captures functional equivariance, allowing for the reuse of skills to achieve new goals. We learn this representation using a metric form of this abstraction, and show its ability to generalize to new goals in simulation manipulation tasks. Further, we prove that this learned representation is sufficient not only for goal-conditioned tasks, but is amenable to any downstream task described by a state-only reward function. Videos can be found at https://sites.google.com/ view/gc-bisimulation .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

27 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Contrastive Difference Predictive Coding cites Reinforcement Learning with Augmented Data
  • Contrastive Difference Predictive Coding cites Offline Reinforcement Learning with Implicit Q-Learning
  • Contrastive Difference Predictive Coding cites CURL: Contrastive Unsupervised Representations for Reinforcement Learning
  • Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels cites Reinforcement Learning with Augmented Data
  • Reinforcement Learning with Augmented Data cites Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
  • Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels cites Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
  • Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels cites CURL: Contrastive Unsupervised Representations for Reinforcement Learning
  • CURL: Contrastive Unsupervised Representations for Reinforcement Learning cites Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
  • Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making cites Contrastive Difference Predictive Coding
  • Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making cites Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View
  • Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making cites What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?
  • Reinforcement Learning with Augmented Data cites Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
  • Reinforcement Learning with Augmented Data cites CURL: Contrastive Unsupervised Representations for Reinforcement Learning
  • CURL: Contrastive Unsupervised Representations for Reinforcement Learning cites Reinforcement Learning with Augmented Data
  • Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View cites Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
  • Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View cites Offline Reinforcement Learning with Implicit Q-Learning
  • Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View cites CURL: Contrastive Unsupervised Representations for Reinforcement Learning
  • CURL: Contrastive Unsupervised Representations for Reinforcement Learning cites Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
  • What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL? cites Offline Reinforcement Learning with Implicit Q-Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers27

  1. Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain AdaptationPeide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi et al.NeurIPS 2022 · 44 citations
  2. Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-MakingVivek Myers, Chongyi Zheng, Anca D. Dragan, Sergey Levine et al.ICML 2024 · 38 citations
  3. What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu et al.ICML 2023 · 35 citations
  4. Contrastive Difference Predictive CodingChongyi Zheng, Ruslan Salakhutdinov, Benjamin EysenbachICLR 2024 · 32 citations
  5. Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of ViewRaj Ghugare, Matthieu Geist, Glen Berseth, Benjamin EysenbachICLR 2024 · 28 citations

Builds on14

  1. Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, Sergey LevineICLR 2022 · 1,402 citations
  2. CURL: Contrastive Unsupervised Representations for Reinforcement LearningMichael Laskin, Aravind Srinivas, Pieter AbbeelICML 2020 · 1,261 citations
  3. Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from PixelsDenis Yarats, Ilya Kostrikov, Rob FergusICLR 2021 · 911 citations
  4. Reinforcement Learning with Augmented DataMichael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto et al.NeurIPS 2020 · 833 citations
  5. Improving Sample Efficiency in Model-Free Reinforcement Learning from ImagesDenis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos et al.AAAI 2021 · 506 citations

Related papers

  1. Building Minimal and Reusable Causal State Abstractions for Reinforcement LearningZizhao Wang, Caroline Wang, Xuesu Xiao, Yuke Zhu et al.AAAI 2024 · 9 citations
  2. Learning Invariant Representations for Reinforcement Learning without ReconstructionAmy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal et al.ICLR 2021 · 77 citations
  3. Hierarchical Goal Abstractions via Learned Subset RelationsFabian Wurzberger, Sebastian Gottwald, Zeqiang Zhang, Daniel A BraunICML 2026
  4. Towards Robust Bisimulation Metric LearningMete Kemertas, Tristan Aumentado-ArmstrongNeurIPS 2021 · 68 citations
  5. Compositional Behavioral Semantics for State Abstraction in Reinforcement LearningYivan Zhang, Ziyan Luo, Manuel BaltieriICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines