Lune

ACL Findings2025

Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation

Chenkai Sun, Denghui Zhang, ChengXiang Zhai, Heng Ji

Abstract

Given the growing influence of language modelbased agents on high-stakes societal decisions, from public policy to healthcare, ensuring their beneficial impact requires understanding the far-reaching implications of their suggestions. We propose a proof-of-concept framework that projects how model-generated advice could propagate through societal systems on a macroscopic scale over time, enabling more robust alignment. To assess the long-term safety awareness of language models, we also introduce a dataset of 100 indirect harm scenarios, testing models' ability to foresee adverse, nonobvious outcomes from seemingly harmless user prompts. Our approach achieves not only over 20% improvement on the new dataset but also an average win rate exceeding 70% against strong baselines on existing safety benchmarks (AdvBench, SafeRLHF, WildGuardMix), suggesting a promising direction for safer agents.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Self-Refine: Iterative Refinement with Self-Feedback cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Self-Refine: Iterative Refinement with Self-Feedback
  • Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
  4. Dream to Control: Learning Behaviors by Latent ImaginationDanijar Hafner, Timothy P. Lillicrap, Jimmy Ba, Mohammad NorouziICLR 2020 · 1,852 citations
  5. Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations

Related papers

  1. When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily LifeXinyue Lou, Jinan Xu, Jingyi Yin, Xiaolong Wang et al.ACL Findings 2026 · 1 citation
  2. SafeToolBench: Pioneering a Prospective Benchmark to Evaluating Tool Utilization Safety in LLMsHongfei Xia, Hongru Wang, Zeming Liu, Qian Yu et al.EMNLP Findings 2025 · 1 citation
  3. Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language ModelsYeonjun In, Wonjoong Kim, Kanghoon Yoon, Sungchul Kim et al.EMNLP Findings 2025 · 1 citation
  4. SafeText: A Benchmark for Exploring Physical Safety in Language ModelsSharon Levy, Emily Allaway, Melanie Subbiah, Lydia B. Chilton et al.EMNLP 2022 · 14 citations
  5. HarDBench: A Benchmark for Draft-Based Co-Authoring Jailbreak Attacks for Safe Human-LLM Collaborative WritingEuntae Kim, Soomin Han, Buru ChangACL 2026
Dusk over the sea between two cliffs drawn in fine vertical lines