Lune

ICML2025Top-tier venue

Safety-Polarized and Prioritized Reinforcement Learning

Ke Fan, Jinpeng Zhang, Xuefeng Zhang, Yunze Wu, Jingyu Cao, Yuan Zhou, Jianzhu Ma

Abstract

Motivated by the first priority of safety in many real-world applications, we propose MAXSAFE, a chance-constrained bi-level optimization framework for safe reinforcement learning. MAXSAFE first minimizes the unsafe probability and then maximizes the return among the safest policies. We provide a tailored Q-learning algorithm for the MAXSAFE objective, featuring a novel learning process for optimal action masks with theoretical convergence guarantees. To enable the application of our algorithm to large-scale experiments, we introduce two key techniques: safety polarization and safety prioritized experience replay. Safety polarization generalizes the optimal action masking by polarizing the Q-function, which assigns low values to unsafe state-action pairs, effectively discouraging their selection. In parallel, safety prioritized experience replay enhances the learning of optimal action masks by prioritizing samples based on temporal-difference (TD) errors derived from our proposed state-action reachability estimation functions. This approach efficiently addresses the challenges posed by sparse cost signals. Experiments on diverse autonomous driving and safe control tasks show that our methods achieve near-maximal safety and an optimal reward-safety trade-off.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model cites Reachability Constrained Reinforcement Learning
  • Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model cites First Order Constrained Optimization in Policy Space
  • Constrained Update Projection Approach to Safe Policy Optimization cites First Order Constrained Optimization in Policy Space
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Episodic Memory-Guided Controllable Experience Synthesis for Reinforcement LearningXiao Ma, Tian Li, Wu-Jun LiICML 2026

Builds on11

  1. First Order Constrained Optimization in Policy SpaceYiming Zhang, Quan Vuong, Keith W. RossNeurIPS 2020 · 238 citations
  2. Constrained Update Projection Approach to Safe Policy OptimizationLong Yang, Jiaming Ji, Juntao Dai, Linrui Zhang et al.NeurIPS 2022 · 95 citations
  3. Neurosymbolic Reinforcement Learning with Formally Verified ExplorationGreg Anderson, Abhinav Verma, Isil Dillig, Swarat ChaudhuriNeurIPS 2020 · 91 citations
  4. Reachability Constrained Reinforcement LearningDongjie Yu, Haitong Ma, Sheng-bo Li, Jianyu ChenICML 2022 · 90 citations
  5. Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion ModelYinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang et al.ICLR 2024 · 72 citations

Related papers

  1. Off-Policy Safe Reinforcement Learning with Cost-Constrained Optimistic ExplorationGuopeng Li, Matthijs T. J. Spaan, Julian F. P. KooijICLR 2026
  2. Iterative Reachability Estimation for Safe Reinforcement LearningMilan Ganai, Zheng Gong, Chenning Yu, Sylvia L. Herbert et al.NeurIPS 2023 · 56 citations
  3. Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPsTao Liu, Ruida Zhou, Dileep Kalathil, Panganamala R. Kumar et al.NeurIPS 2021 · 110 citations
  4. Enhancing Efficiency of Safe Reinforcement Learning via Sample ManipulationShangding Gu, Laixi Shi, Yuhao Ding, Alois Knoll et al.NeurIPS 2024 · 14 citations
  5. Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement LearningChenglin Li, Guangchun Ruan, Hua GengAAAI 2025
Dusk over the sea between two cliffs drawn in fine vertical lines