ACL Findings2025
Red-Teaming LLM Multi-Agent Systems via Communication Attacks
Pengfei He, Yuping Lin, Shen Dong, Han Xu, Yue Xing, Hui Liu
Abstract
Large Language Model-based Multi-Agent Systems (LLM-MAS) have revolutionized complex problem-solving capability by enabling sophisticated agent collaboration through message-based communications. While the communication framework is crucial for agent coordination, it also introduces a critical yet unexplored security vulnerability. In this work, we introduce Agent-in-the-Middle (AiTM), a novel attack that exploits the fundamental communication mechanisms in LLM-MAS by intercepting and manipulating inter-agent messages. Unlike existing attacks that compromise individual agents, AiTM demonstrates how an adversary can compromise entire multi-agent systems by only manipulating the messages passing between agents. To enable the attack under the challenges of limited control and rolerestricted communication format, we develop an LLM-powered adversarial agent with a reflection mechanism that generates contextuallyaware malicious instructions. Our comprehensive evaluation across various frameworks, communication structures, and real-world applications demonstrates that LLM-MAS is vulnerable to communication-based attacks, highlighting the need for robust security measures in multi-agent systems.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
19 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
- Generative Agents: Interactive Simulacra of Human Behavior cites Language Models are Few-Shot Learners
- Generative Agents: Interactive Simulacra of Human Behavior cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling cites Measuring Massive Multitask Language Understanding
- GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling cites Improving Factuality and Reasoning in Language Models through Multiagent Debate
- GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Goal-Aware Identification and Rectification of Misinformation in Multi-Agent Systems cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Measuring Massive Multitask Language Understanding
- Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers19
- SAGA: A Security Architecture for Governing AI Agentic SystemsGeorgios Syros, Anshuman Suri, Jacob Ginesin, Cristina Nita-Rotaru et al.NDSS 2026 · 63 citations
- GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph ModelingJialong Zhou, Lichao Wang, Xiao YangNeurIPS 2025 · 40 citations
- TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM SystemsIshan Kavathekar, Hemang Jain, Ameya Rathod, Ponnurangam Kumaraguru et al.ACL 2026 · 16 citations
- Goal-Aware Identification and Rectification of Misinformation in Multi-Agent SystemsZherui Li, Yan Mi, Zhenhong Zhou, Houcheng Jiang et al.ICLR 2026 · 10 citations
- Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM JudgesHamin Koo, Minseon Kim, Jaehyung KimICLR 2026 · 4 citations
Builds on17
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Generative Agents: Interactive Simulacra of Human BehaviorJoon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris et al.UIST 2023 · 1,882 citations
- Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations
Related papers
- Attack the Messages, Not the Agents: A Multi-round Adaptive Stealthy Tampering Framework for LLM-MASBingyu Yan, Xiaoming Zhang, Ziyi Zhou, Chaozhuo Li et al.AAAI 2026 · 1 citation
- MASTER: Multi-Agent Security Through Exploration of Roles and Topological Structures - A Comprehensive FrameworkYifan Zhu, Chao Zhang, Xin Shi, Xueqiao Zhang et al.EMNLP Findings 2025
- When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent SystemsHaowen Xu, Xue Tan, Lei Ma, Zhihao Zhang et al.ICML 2026
- ResMAS: Resilience Optimization in LLM-based Multi-agent SystemsZhilun Zhou, Zihan Liu, Jiahe Liu, Qingyu Shao et al.AAAI 2026 · 2 citations
- G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent SystemsShilong Wang, Guibin Zhang, Miao Yu, Guancheng Wan et al.ACL 2025 · 37 citations
