Lune

ACL Findings2026

MalURLBench: A Benchmark Evaluating Agents' Vulnerabilities When Processing Web URLs

Dezhang Kong, Zhuxi Wu, Shiqi Liu, Zhicheng Tan, Kuichen Lu, Minghao Li, Qichen Liu, Shengyu Chu, Zhenhua Xu, Xuan Liu, Meng Han

Abstract

LLM-based web agents have become increasingly popular for their utility in daily life and work. However, they exhibit critical vulnerabilities when processing malicious URLs: accepting a disguised malicious URL enables subsequent access to unsafe webpages, which can cause severe damage to service providers and users. Despite this risk, no benchmark currently targets this emerging threat. To address this gap, we propose MalURLBench, the first benchmark for evaluating LLMs' vulnerabilities to malicious URLs. MalURLBench contains 61,845 attack instances spanning 10 real-world scenarios and 7 categories of real malicious websites. Experiments with 12 popular LLMs reveal that existing models struggle to detect elaborately disguised malicious URLs. We further identify and analyze key factors that impact attack success rates and propose URLGuard, a lightweight defense module. We believe this work will provide a foundational resource for advancing the security of web agents. Our code is available at https: //github.com/JiangYingEr/MalURLBench.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization cites Automatic Prompt Optimization with "Gradient Descent" and Beam Search
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  2. GPT-4V(ision) is a Generalist Web Agent, if GroundedBoyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun et al.ICML 2024 · 496 citations
  3. Automatic Prompt Optimization with "Gradient Descent" and Beam SearchReid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee et al.EMNLP 2023 · 137 citations
  4. Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt OptimizationXingchen Wan, Ruoxi Sun, Hootan Nakhost, Sercan Ö. ArikNeurIPS 2024 · 35 citations
  5. RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use AgentsJingyi Yang, Shuai Shao, Dongrui Liu, Jing ShaoNeurIPS 2025 · 33 citations

Related papers

  1. WebCloak: Characterizing and Mitigating Threats From LLM-Driven Web Agents as Intelligent ScrapersXinfeng Li, Tianze Qiu, Yingbin Jin, Lixu Wang et al.S&P 2026 · 13 citations
  2. CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application VulnerabilitiesYuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li et al.ICML 2025 · 1 citation
  3. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based AgentsHanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao et al.ICLR 2025
  4. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model AgentsQiusi Zhan, Zhixiang Liang, Zifan Ying, Daniel KangACL Findings 2024 · 147 citations
  5. AutoAdvExBench: Benchmarking Autonomous Exploitation of Adversarial Example DefensesNicholas Carlini, Edoardo Debenedetti, Javier Rando, Milad Nasr et al.ICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines