ACL Findings2025
Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation
Chenkai Sun, Denghui Zhang, ChengXiang Zhai, Heng Ji
Abstract
Given the growing influence of language modelbased agents on high-stakes societal decisions, from public policy to healthcare, ensuring their beneficial impact requires understanding the far-reaching implications of their suggestions. We propose a proof-of-concept framework that projects how model-generated advice could propagate through societal systems on a macroscopic scale over time, enabling more robust alignment. To assess the long-term safety awareness of language models, we also introduce a dataset of 100 indirect harm scenarios, testing models' ability to foresee adverse, nonobvious outcomes from seemingly harmless user prompts. Our approach achieves not only over 20% improvement on the new dataset but also an average win rate exceeding 70% against strong baselines on existing safety benchmarks (AdvBench, SafeRLHF, WildGuardMix), suggesting a promising direction for safer agents.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Self-Refine: Iterative Refinement with Self-Feedback cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Self-Refine: Iterative Refinement with Self-Feedback
- Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds on11
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Dream to Control: Learning Behaviors by Latent ImaginationDanijar Hafner, Timothy P. Lillicrap, Jimmy Ba, Mohammad NorouziICLR 2020 · 1,852 citations
- Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations
Related papers
- When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily LifeXinyue Lou, Jinan Xu, Jingyi Yin, Xiaolong Wang et al.ACL Findings 2026 · 1 citation
- SafeToolBench: Pioneering a Prospective Benchmark to Evaluating Tool Utilization Safety in LLMsHongfei Xia, Hongru Wang, Zeming Liu, Qian Yu et al.EMNLP Findings 2025 · 1 citation
- Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language ModelsYeonjun In, Wonjoong Kim, Kanghoon Yoon, Sungchul Kim et al.EMNLP Findings 2025 · 1 citation
- SafeText: A Benchmark for Exploring Physical Safety in Language ModelsSharon Levy, Emily Allaway, Melanie Subbiah, Lydia B. Chilton et al.EMNLP 2022 · 14 citations
- HarDBench: A Benchmark for Draft-Based Co-Authoring Jailbreak Attacks for Safe Human-LLM Collaborative WritingEuntae Kim, Soomin Han, Buru ChangACL 2026
