Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning
Hanlin Yang, Jian Yao, Weiming Liu, Qing Wang, Hanmin Qin, Hansheng Kong, Kirk Tang, Jiechao Xiong, Chao Yu, Kai Li, Junliang Xing, Hongwu Chen
Abstract
Recovering a spectrum of diverse policies from a set of expert trajectories is an important research topic in imitation learning. After determining a latent style for a trajectory, previous diverse policies recovering methods usually employ a vanilla behavioral cloning learning objective conditioned on the latent style, treating each state-action pair in the trajectory with equal importance. Based on an observation that in many scenarios, behavioral styles are often highly relevant with only a subset of state-action pairs, this paper presents a new principled method in diverse polices recovery. In particular, after inferring or assigning a latent style for a trajectory, we enhance the vanilla behavioral cloning by incorporating a weighting mechanism based on pointwise mutual information. This additional weighting reflects the significance of each state-action pair's contribution to learning the style, thus allowing our method to focus on state-action pairs most representative of that style. We provide theoretical justifications for our new objective, and extensive empirical evaluations confirm the effectiveness of our method in recovering diverse policies from expert data.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Learning Calibratable Policies using Programmatic Style-Consistency cites Dynamics-Aware Unsupervised Discovery of Skills
- Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment cites Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset
- Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment cites Learning Calibratable Policies using Programmatic Style-Consistency
- Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment cites Offline Reinforcement Learning with Implicit Q-Learning
- Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills cites Dynamics-Aware Unsupervised Discovery of Skills
Cited by top-tier papers2
Builds on6
- Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, Sergey LevineICLR 2022 · 1,402 citations
- Dynamics-Aware Unsupervised Discovery of SkillsArchit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar et al.ICLR 2020 · 475 citations
- Explore, Discover and Learn: Unsupervised Discovery of State-Covering SkillsVictor Campos, Alexander Trott, Caiming Xiong, Richard Socher et al.ICML 2020 · 178 citations
- Atari-HEAD: Atari Human Eye-Tracking and Demonstration DatasetRuohan Zhang, Calen Walshe, Zhuode Liu, Lin Guan et al.AAAI 2020 · 77 citations
- Learning Calibratable Policies using Programmatic Style-ConsistencyEric Zhan, Albert Tseng, Yisong Yue, Adith Swaminathan et al.ICML 2020 · 21 citations
Related papers
- Imitation Beyond Expectation Using Pluralistic Stochastic DominanceAli Farajzadeh, Danyal Saeed, Syed M. Abbas, Rushit N. Shah et al.NeurIPS 2025 · 2 citations
- Stylized Offline Reinforcement Learning: Extracting Diverse High-Quality Behaviors from Heterogeneous DatasetsYihuan Mao, Chengjie Wu, Xi Chen, Hao Hu et al.ICLR 2024 · 9 citations
- Information Maximizing Curriculum: A Curriculum-Based Approach for Learning Versatile SkillsDenis Blessing, Onur Celik, Xiaogang Jia, Moritz Reuss et al.NeurIPS 2023 · 19 citations
- Causal Imitation Learning With Unobserved ConfoundersJunzhe Zhang, Daniel Kumor, Elias BareinboimNeurIPS 2020 · 86 citations
- Learning Distinguishable Trajectory Representation with Contrastive LossTianxu Li, Kun Zhu, Juan Li, Yang ZhangNeurIPS 2024 · 5 citations
