Generalizing Reinforcement Learning through Fusing Self-Supervised Learning into Intrinsic Motivation
Keyu Wu, Min Wu, Zhenghua Chen, Yuecong Xu, Xiaoli Li
Abstract
Despite the great potential of reinforcement learning (RL) in solving complex decision-making problems, generalization remains one of its key challenges, leading to difficulty in deploying learned RL policies to new environments. In this paper, we propose to improve the generalization of RL algorithms through fusing Self-supervised learning into Intrinsic Motivation (SIM). Specifically, SIM boosts representation learning through driving the cross-correlation matrix between the embeddings of augmented and non-augmented samples close to the identity matrix. This aims to increase the similarity between the embedding vectors of a sample and its augmented version while minimizing the redundancy between the components of these vectors. Meanwhile, the redundancy reduction based self-supervised loss is converted to an intrinsic reward to further improve generalization in RL via an auxiliary objective. As a general paradigm, SIM can be implemented on top of any RL algorithm. Extensive evaluations have been performed on a diversity of tasks. Experimental results demonstrate that SIM consistently outperforms the state-of-the-art methods and exhibits superior generalization capability and sample efficiency.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return Prediction cites CURL: Contrastive Unsupervised Representations for Reinforcement Learning
- What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return Prediction cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- CURL: Contrastive Unsupervised Representations for Reinforcement Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites What Makes for Good Views for Contrastive Learning?
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction cites A Simple Framework for Contrastive Learning of Visual Representations
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- What Makes for Good Views for Contrastive Learning? cites A Simple Framework for Contrastive Learning of Visual Representations
- What Makes for Good Views for Contrastive Learning? cites CURL: Contrastive Unsupervised Representations for Reinforcement Learning
Cited by top-tier papers2
- What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return PredictionShuo Wang, Zhihao Wu, Xiaobo Hu, Jinwen Wang et al.AAAI 2024 · 18 citations
- Analyzing Generalization in Policy Networks: A Case Study with the Double-Integrator SystemRuining Zhang, Haoran Han, Maolong Lv, Qisong Yang et al.AAAI 2024 · 5 citations
Builds on10
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
- Barlow Twins: Self-Supervised Learning via Redundancy ReductionJure Zbontar, Li Jing, Ishan Misra, Yann LeCun et al.ICML 2021 · 2,942 citations
- What Makes for Good Views for Contrastive Learning?Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan et al.NeurIPS 2020 · 1,631 citations
- CURL: Contrastive Unsupervised Representations for Reinforcement LearningMichael Laskin, Aravind Srinivas, Pieter AbbeelICML 2020 · 1,261 citations
Related papers
- Self-Supervised Reinforcement Learning that Transfers using Random FeaturesBoyuan Chen, Chuning Zhu, Pulkit Agrawal, Kaiqing Zhang et al.NeurIPS 2023 · 16 citations
- Dynamics-Aware EmbeddingsWilliam F. Whitney, Rajat Agarwal, Kyunghyun Cho, Abhinav GuptaICLR 2020
- Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline DataShilong Deng, Zetao Zheng, Hongcai He, Paul Weng et al.AAAI 2025
- Curious Representation Learning for Embodied IntelligenceYilun Du, Chuang Gan, Phillip IsolaICCV 2021 · 50 citations
- Improving Zero-Shot Generalization in Offline Reinforcement Learning using Generalized Similarity FunctionsBogdan Mazoure, Ilya Kostrikov, Ofir Nachum, Jonathan TompsonNeurIPS 2022 · 29 citations
