Deep Implicit Imitation Reinforcement Learning in Heterogeneous Action Settings
Iason Chrysomallis, Georgios Chalkiadakis, Ioannis Papamichail, Markos Papageorgiou
Abstract
Implicit imitation reinforcement learning (IIRL) is a framework that aims to aid a trainee agent's learning process via observing the state transitions of a mentor, but without access to the latter's action information. Standard IIRL assumes a shared Markov decision process (MDP) between the mentor and trainee, consequently implying an identical action space. This restriction imposes limitations on the applicability of implicit imitation frameworks in real-life scenarios where, possibly due to variations in physical characteristics, the mentor agent may possess distinct own actions, thereby creating a heterogeneous action setting. In this work, we extend the deep implicit imitation Q-networks (DIIQN) method-an online, model-free, deep RL algorithm for implicit imitation-to allow for heterogeneous action sets between mentor and trainee agents. Equipped with our heterogeneous actions DIIQN (HA-DIIQN) method, a trainee agent can harvest the benefits of IIRL even in heterogeneous action settings, achieving accelerated learning and outperforming non-optimal mentor agents.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Related papers
- State-only Imitation with Transition Dynamics MismatchTanmay Gangwani, Jian PengICLR 2020 · 56 citations
- Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation LearningXin-Qiang Cai, Yao-Xiang Ding, Zi-Xuan Chen, Yuan Jiang et al.ICLR 2023 · 2 citations
- Multi-Agent Interactions Modeling with Correlated PoliciesMinghuan Liu, Ming Zhou, Weinan Zhang, Yuzheng Zhuang et al.ICLR 2020 · 22 citations
- Deep Demonstration Tracing: Learning Generalizable Imitator Policy for Runtime Imitation from a Single DemonstrationXiong-Hui Chen, Junyin Ye, Hang Zhao, Yi-Chen Li et al.ICML 2024 · 2 citations
- SEABO: A Simple Search-Based Method for Offline Imitation LearningJiafei Lyu, Xiaoteng Ma, Le Wan, Runze Liu et al.ICLR 2024 · 17 citations
