Lune

ICLR2025Top-tier venue

Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, Yongfeng Zhang

Abstract

Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLMbased agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and 11 corresponding defenses across 13 LLM backbones. Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. We also introduce a new metric to evaluate the agents' capability to balance utility and security. Our code can be found at https: //github.com/agiresearch/ASB . Furthermore, ASB explores the vulnerabilities in agents performing tasks in diverse settings. Specifically, ASB evaluates across 10 task scenarios, 10 corresponding agents, and over 400 tools, including both normal and attack tools, and 400 tasks, divided into aggressive and non-aggressive types. The aggressive tasks assess the agent's refusal rate in response to risky or aggressive instructions. Our key contributions are summarized as follows: ① We design and develop Agent Security Bench (ASB), the first comprehensive benchmark including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools and tasks for evaluating the security of LLM-based agents against numerous attacks and defense strategies. ② We propose a novel PoT Backdoor Attack, which embeds hidden instructions into the system prompt, exploiting the agent's planning process to achieve high attack success rates. ③ We formalize and categorize various adversarial threats targeting key components of LLM agents, including DPI, IPI, Memory Poisoning Attacks, PoT Backdoor Attacks, and Mixed Attacks, covering vulnerabilities in system prompt definition, user prompt handling, memory retrieval, and tool usage. ④ We benchmark 27 different types of attacks and defenses on ASB across 13 LLM backbones with 7 metrics, demonstrating that LLM-based agents are vulnerable to the attacks, with the highest average attack success rates exceeding 84.30%. In contrast, existing defenses are often ineffective. Our work highlights the need for stronger defenses to protect LLM agents from sophisticated adversarial techniques. ⑤ We introduce the Net Resilient Performance (NRP) metric to assess the agents' balance between utility and security and emphasize the importance of performance testing on ASB for selecting suitable backbones for agent applications.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

51 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Gorilla: Large Language Model Connected with Massive APIs cites Large Language Models are Zero-Shot Reasoners
  • Gorilla: Large Language Model Connected with Massive APIs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents cites ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
  • AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents cites Large Language Models are Zero-Shot Reasoners
  • AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents cites Large Language Models are Zero-Shot Reasoners
  • DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers51

  1. AgentAuditor: Human-level Safety and Security Evaluation for LLM AgentsHanjun Luo, Shenyu Dai, Chiming Ni, Xinfeng Li et al.NeurIPS 2025 · 98 citations
  2. ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web AgentsIdo Levy, Ben wiesel, Sami Marreed, Alon Oved et al.ICLR 2026 · 78 citations
  3. DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM AgentsHao Li, Xiaogeng Liu, Hung-Chun Chiu, Dianqi Li et al.NeurIPS 2025 · 76 citations
  4. ACE: A Security Architecture for LLM-Integrated App SystemsEvan Li, Tushin Mallick, Evan Rose, William K. Robertson et al.NDSS 2026 · 57 citations
  5. MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM AgentsDongsen Zhang, Zekun Li, Xu Luo, Xuannan Liu et al.ICLR 2026 · 47 citations

Builds on26

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  3. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  4. Gorilla: Large Language Model Connected with Massive APIsShishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. GonzalezNeurIPS 2024 · 1,715 citations
  5. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied AgentsWenlong Huang, Pieter Abbeel, Deepak Pathak, Igor MordatchICML 2022 · 1,539 citations

Related papers

  1. AgentLAB: Benchmarking LLM Agents against Long-Horizon AttacksTanqiu Jiang, Yuhui Wang, Jiacheng Liang, Ting WangICML 2026 · 21 citations
  2. MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP ServersXuanjun Zong, Zhiqi Shen, Lei Wang, Yunshi Lan et al.ICLR 2026 · 34 citations
  3. SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security TasksHwiwon Lee, Ziqi Zhang, Hanxiao Lu, Lingming ZhangNeurIPS 2025 · 86 citations
  4. MalURLBench: A Benchmark Evaluating Agents' Vulnerabilities When Processing Web URLsDezhang Kong, Zhuxi Wu, Shiqi Liu, Zhicheng Tan et al.ACL Findings 2026
  5. BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based AgentsYunhao Feng, Yige Li, Yutao Wu, Yingshui Tan et al.ACL Findings 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines