Toward Exploratory Inverse Constraint Inference with Generative Diffusion Verifiers
Abstract
An important prerequisite for safe control is aligning the policy with the underlying constraints in the environment. In many real-world applications, due to the difficulty of manually specifying these constraints, existing works have proposed recovering constraints from expert demonstrations by solving the Inverse Constraint Learning (ICL) problem. However, ICL is inherently ill-posed, as multiple constraints can equivalently explain the experts' preferences, making the optimal solutions not uniquely identifiable. In this work, instead of focusing solely on a single constraint, we propose the novel approach of Exploratory ICL (ExICL). The goal of ExICL is to recover a diverse set of feasible constraints, thereby providing practitioners the flexibility to select the most appropriate constraint based on the practical needs of deployment. To achieve this goal, we design a generative diffusion verifier that guides the trajectory generation process using the probabilistic representation of an optimal constrained policy. By comparing these decisions with those made by expert agents, we can efficiently verify a candidate constraint. Driven by the verification feedback, ExICL implements an exploratory constraint update mechanism that strategically facilitates diversity within the collection of feasible constraints. Our empirical results demonstrate that ExICL can seamlessly and reliably generalize across different tasks and environments. The code is available at https://github.com/ZhaoRunyi/ExICL.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Planning with Diffusion for Flexible Behavior Synthesis cites Denoising Diffusion Probabilistic Models
- AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners cites Planning with Diffusion for Flexible Behavior Synthesis
- AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners cites Denoising Diffusion Probabilistic Models
- Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning cites Planning with Diffusion for Flexible Behavior Synthesis
- Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning cites AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners
- Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning cites Denoising Diffusion Probabilistic Models
Cited by top-tier papers1
Builds on29
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Planning with Diffusion for Flexible Behavior SynthesisMichael Janner, Yilun Du, Joshua B. Tenenbaum, Sergey LevineICML 2022 · 1,115 citations
- IQ-Learn: Inverse soft-Q Learning for ImitationDivyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song et al.NeurIPS 2021 · 271 citations
- AdaptDiffuser: Diffusion Models as Adaptive Self-evolving PlannersZhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni et al.ICML 2023 · 165 citations
- Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement LearningHaoran He, Chenjia Bai, Kang Xu, Zhuoran Yang et al.NeurIPS 2023 · 165 citations
Related papers
- Learning Shared Safety Constraints from Multi-task DemonstrationsKonwoo Kim, Gokul Swamy, Zuxin Liu, Ding Zhao et al.NeurIPS 2023 · 31 citations
- Confidence Aware Inverse Constrained Reinforcement LearningSriram Ganapathi Subramanian, Guiliang Liu, Mohammed Elmahgiubi, Kasra Rezaee et al.ICML 2024 · 5 citations
- Multi-Modal Inverse Constrained Reinforcement Learning from a Mixture of DemonstrationsGuanren Qiao, Guiliang Liu, Pascal Poupart, Zhiqiang XuNeurIPS 2023 · 28 citations
- Benchmarking Constraint Inference in Inverse Reinforcement LearningGuiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee et al.ICLR 2023 · 2 citations
- Learning Constraints from Offline Demonstrations via Superior Distribution Correction EstimationGuorui Quan, Zhiqiang Xu, Guiliang LiuICML 2024 · 4 citations
