Lune

ACL Findings2026

AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models

Yixu Wang, Xin Wang, Yang Yao, Xinyuan Li, Xibang Yang, Yan Teng, Xingjun Ma, Yingchun Wang

Abstract

The rapid integration of Large Language Models (LLMs) into high-stakes domains necessitates reliable safety and compliance evaluation. However, existing static benchmarks are ill-equipped to address the dynamic nature of AI risks and evolving regulations, creating a critical safety gap. This paper introduces a new paradigm of agentic safety evaluation, reframing evaluation as a continuous and selfevolving process rather than a one-time audit. We then propose a novel multi-agent framework AgenticEval, which autonomously ingests unstructured policy documents to generate and perpetually evolve a comprehensive safety benchmark. AgenticEval leverages a synergistic pipeline of specialized agents and incorporates a Self-evolving Evaluation loop, where the system learns from evaluation results to craft progressively more sophisticated and targeted test cases. Our experiments demonstrate the effectiveness of AgenticEval, showing a consistent decline in model safety as the evaluation hardens. For instance, GPT-5's safety rate on the EU AI Act drops from 72.50% to 36.36% over successive iterations. These findings reveal the limitations of static assessments and highlight our framework's ability to uncover deep vulnerabilities missed by traditional methods, underscoring the urgent need for dynamic evaluation ecosystems to ensure the safe and responsible deployment of advanced AI.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration cites AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on7

  1. ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based EvaluationJingnan Zheng, Han Wang, An Zhang, Tai D. Nguyen et al.NeurIPS 2024 · 61 citations
  2. AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack IntegrationAndy Zhou, Kevin Wu, Francesco Pinto, Zhaorun Chen et al.NeurIPS 2025 · 46 citations
  3. Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative ModelsFan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao et al.ACL 2025 · 32 citations
  4. RedDebate: Safer Responses Through Multi-Agent Red Teaming DebatesAli Asad, Stephen Obadinma, Radin Shayanfar, Xiaodan ZhuICML 2026 · 7 citations
  5. AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMsXiaogeng Liu, Peiran Li, G. Edward Suh, Yevgeniy Vorobeychik et al.ICLR 2025

Related papers

  1. SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security TasksHwiwon Lee, Ziqi Zhang, Hanxiao Lu, Lingming ZhangNeurIPS 2025 · 86 citations
  2. Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language ModelsYeonjun In, Wonjoong Kim, Kanghoon Yoon, Sungchul Kim et al.EMNLP Findings 2025 · 1 citation
  3. Inverting the Shield: Systematically Generating Safety Tests from Policy SpecificationsXiaoyue Lu, Xianglin Yang, Haijun Liu, Jiahao Liu et al.ACL 2026
  4. AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety DetectionWeidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee et al.ACL 2025 · 41 citations
  5. LongSafety: Evaluating Long-Context Safety of Large Language ModelsYida Lu, Jiale Cheng, Zhexin Zhang, Shiyao Cui et al.ACL 2025 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines