Lune

ICML2021顶会

Safe Reinforcement Learning Using Advantage-Based Intervention

Nolan Wagener, Byron Boots, Ching-An Cheng

摘要

Many sequential decision problems involve finding a policy that maximizes total reward while obeying safety constraints. Although much recent research has focused on the development of safe reinforcement learning (RL) algorithms that produce a safe policy after training, ensuring safety during training as well remains an open problem. A fundamental challenge is performing exploration while still satisfying constraints in an unknown Markov decision process (MDP). In this work, we address this problem for the chanceconstrained setting. We propose a new algorithm, SAILR, that uses an intervention mechanism based on advantage functions to keep the agent safe throughout training and optimizes the agent's policy using off-the-shelf RL algorithms designed for unconstrained MDPs. Our method comes with strong guarantees on safety during both training and deployment (i.e., after training and without the intervention mechanism) and policy performance compared to the optimal safetyconstrained policy. In our experiments, we show that SAILR violates constraints far less during training than standard safe RL and constrained MDP approaches and converges to a wellperforming policy that can be deployed safely without intervention. Our code is available at https://github.com/nolanwagener/safe_rl . * be its state-action value function, and V * be its state value function. be a subset of admissible intervention rules with a threshold of zero and average as the advantage function of the optimal policy. For some intervention rule G ∈ G 0 and policy π, let π = G(π). Then, the inequality A(s, a) ≥ A * (s, a) holds for all a ∈ A almost surely over the distribution d π (s). Proof. First, we show by induction that running π starting from d 0 results in the agent staying in the subset S G = s ∈ S : For t = 0, consider some s 0 ∼ d 0 . We observe from admissibility of G and Proposition 2 that Now suppose the agent is in S G with probability one at some time step t. Consider some s t ∼ d π t (observing that s t ∈ S G ). We assume that s t ∈ S safe (otherwise, the below is trivially true as there is no intervention outside S safe ). By Lemma 4 and admissibility, we can derive: where the second and fourth equalities are due to s t ∈ S safe , and the third equality is due to s t ∈ S G . Notice also, since s t ∈ S safe , we have Therefore, combining the two inequalities above, we have Since Q(s, a) ≥ V * (s) on S × A, by the same argument we made for s 0 , we conclude Q(s t+1 , µ) = V * (s t+1 ) with probability one. Therefore, the agent stays in the subset S G . With this property in mind, let s ∼ d π . Then the following holds for all a ∈ A: where the second equality is due to Q Proposition 4. Let π * be an optimal policy for M, Q * be its state-action value function, and V * be its state value function. π * , 0) ∈ G 0 . Consider arbitrary G ∈ G 0 and policy π. Let M and M * be the absorbing MDPs induced by G and G * , respectively, and let d π and d * ,π be their state-action distributions of π. Then, Supp S×A ( d π ) ⊆ Supp S×A ( d * ,π ), where Supp S×A (d) denotes the support of a distribution d when restricted on S × A.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 15 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • Constrained Decision Transformer for Offline Safe Reinforcement Learning 引用了 Constrained Variational Policy Optimization for Safe Reinforcement Learning
  • When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning 引用了 Safe Reinforcement Learning via Curriculum Induction
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper15

  1. Constrained Variational Policy Optimization for Safe Reinforcement LearningZuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu 等ICML 2022 · 112 次引用
  2. Constrained Decision Transformer for Offline Safe Reinforcement LearningZuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen 等ICML 2023 · 82 次引用
  3. Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic EnvironmentsYixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang 等ICML 2023 · 81 次引用
  4. When to Ask for Help: Proactive Interventions in Autonomous Reinforcement LearningAnnie Xie, Fahim Tajwar, Archit Sharma, Chelsea FinnNeurIPS 2022 · 30 次引用
  5. Maximum State Entropy Exploration using Predecessor and Successor RepresentationsArnav Kumar Jain, Lucas Lehnert, Irina Rish, Glen BersethNeurIPS 2023 · 27 次引用

它引用的 Paper7

  1. On Gradient Descent Ascent for Nonconvex-Concave Minimax ProblemsTianyi Lin, Chi Jin, Michael I. JordanICML 2020 · 587 次引用
  2. Safe Reinforcement Learning via Curriculum InductionMatteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause 等NeurIPS 2020 · 109 次引用
  3. Neurosymbolic Reinforcement Learning with Formally Verified ExplorationGreg Anderson, Abhinav Verma, Isil Dillig, Swarat ChaudhuriNeurIPS 2020 · 91 次引用
  4. Upper Confidence Primal-Dual Reinforcement Learning for CMDP with Adversarial LossShuang Qiu, Xiaohan Wei, Zhuoran Yang, Jieping Ye 等NeurIPS 2020 · 65 次引用
  5. Policy Improvement via Imitation of Multiple OraclesChing-An Cheng, Andrey Kolobov, Alekh AgarwalNeurIPS 2020 · 36 次引用

相关 Paper

  1. Constrained Markov Decision Processes via Backward Value FunctionsHarsh Satija, Philip Amortila, Joelle PineauICML 2020 · 58 次引用
  2. Enhancing Safe Exploration Using Safety State AugmentationAivar Sootla, Alexander I. Cowen-Rivers, Jun Wang, Haitham Bou-AmmarNeurIPS 2022 · 23 次引用
  3. DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement LearningArchana Bura, Aria HasanzadeZonuzy, Dileep Kalathil, Srinivas Shakkottai 等NeurIPS 2022 · 48 次引用
  4. SafeMPO: Constrained Reinforcement Learning with Probabilistic Incremental ImprovementAlexander Mattick, Dominik Seuß, Christopher MutschlerICLR 2026
  5. Flipping-based Policy for Chance-Constrained Markov Decision ProcessesXun Shen, Shuo Jiang, Akifumi Wachi, Kazumune Hashimoto 等NeurIPS 2024 · 5 次引用
黄昏的海面,两侧是细线勾勒的悬崖