Unsupervised Zero-Shot Reinforcement Learning via Dual-Value Forward-Backward Representation
Jingbo Sun, Songjun Tu, Qichao Zhang, Haoran Li, Xin Liu, Yaran Chen, Ke Chen, Dongbin Zhao
Abstract
Online unsupervised reinforcement learning (URL) can discover diverse skills via reward-free pre-training and exhibits impressive downstream task adaptation abilities through further fine-tuning. However, online URL methods face challenges in achieving zero-shot generalization, i.e., directly applying pre-trained policies to downstream tasks without additional planning or learning. In this paper, we propose a novel Dual-Value Forward-Backward representation (DVFB) framework with a contrastive entropy intrinsic reward to achieve both zero-shot generalization and fine-tuning adaptation in online URL. On the one hand, we demonstrate that poor exploration in forward-backward representations can lead to limited data diversity in online URL, impairing successor measures, and ultimately constraining generalization ability. To address this issue, the DVFB framework learns successor measures through a skill value function while promoting data diversity through an exploration value function, thus enabling zero-shot generalization. On the other hand, and somewhat surprisingly, by employing a straightforward dual-value finetuning scheme combined with a reward mapping technique, the pre-trained policy further enhances its performance through fine-tuning on downstream tasks. Through extensive experiments, DVFB demonstrates both superior zero-shot generalization (outperforming on all 12 tasks) and fine-tuning adaptation (leading on 10 out of 12 tasks) abilities, surpassing state-of-the-art (SOTA) URL methods.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Mastering Atari with Discrete World Models cites A Simple Framework for Contrastive Learning of Visual Representations
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations cites Mastering Atari with Discrete World Models
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations cites Decoupling Representation Learning from Reinforcement Learning
- Learning and Planning Multi-Agent Tasks via an MoE-based World Model cites Mastering Atari with Discrete World Models
- Decoupling Representation Learning from Reinforcement Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Mastering Atari Games with Limited Data cites Mastering Atari with Discrete World Models
- Mastering Atari Games with Limited Data cites A Simple Framework for Contrastive Learning of Visual Representations
Cited by top-tier papers6
- Learning and Planning Multi-Agent Tasks via an MoE-based World ModelZijie Zhao, Zhongyue Zhao, Kaixuan Xu, Yuqian Fu et al.NeurIPS 2025 · 12 citations
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent RepresentationsXin Liu, Haoran Li, Dongbin ZhaoNeurIPS 2025 · 5 citations
- Equilibrium Policy Generalization: A Reinforcement Learning Framework for Cross-Graph Zero-Shot Generalization in Pursuit-Evasion GamesRunyu Lu, Peng Zhang, Ruochuan Shi, Yuanheng Zhu et al.NeurIPS 2025 · 3 citations
- A Reward-Free Viewpoint on Multi-Objective Reinforcement LearningYing-Tu Chen, Wei Hung, Bing-Shu Wu, Zhang-Wei Hong et al.ICLR 2026 · 2 citations
- Task-Aware Exploration via a Predictive Bisimulation MetricDayang Liang, Ruihan LIU, Lipeng Wan, Yunlong Liu et al.ICML 2026 · 1 citation
Builds on24
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Mastering Atari with Discrete World ModelsDanijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, Jimmy BaICLR 2021 · 1,170 citations
- Dynamics-Aware Unsupervised Discovery of SkillsArchit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar et al.ICLR 2020 · 475 citations
- Decoupling Representation Learning from Reinforcement LearningAdam Stooke, Kimin Lee, Pieter Abbeel, Michael LaskinICML 2021 · 389 citations
- Mastering Atari Games with Limited DataWeirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel et al.NeurIPS 2021 · 345 citations
Related papers
- Does Zero-Shot Reinforcement Learning Exist?Ahmed Touati, Jérémy Rapin, Yann OllivierICLR 2023 · 2 citations
- Unsupervised Zero-Shot Reinforcement Learning via Functional Reward EncodingsKevin Frans, Seohong Park, Pieter Abbeel, Sergey LevineICML 2024 · 26 citations
- Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement LearningJingbo Sun, Qichao Zhang, Songjun Tu, Xing Fang et al.CVPR 2026 · 1 citation
- Exploratory Diffusion Model for Unsupervised Reinforcement LearningChengyang Ying, Huayu Chen, Xinning Zhou, Zhongkai Hao et al.ICLR 2026 · 4 citations
- VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-TrainingYecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani et al.ICLR 2023 · 35 citations
