ACL Findings2025
CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents
Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Anjie Yang, Zhaoxuan Jin
Abstract
The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks for MLM agents in interactive environments are limited by their focus on a single environment, lack of detailed and generalized evaluation methods, and the complexities of constructing tasks and evaluators. To overcome these limitations, we introduce CRAB, the first cross-environment agent benchmark framework, incorporating a graphbased fine-grained evaluation method and an efficient task generation method. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. Leveraging CRAB, we develope CRAB Benchmark-v0 comprising 120 tasks in computer desktop and mobile phone environments. We evaluated 6 advanced MLMs using different single and multi-agent system configurations on this benchmark. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
16 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- UINavBench: A Framework for Comprehensive Evaluation of Interactive Digital Agents cites Language Models can Solve Computer Tasks
- AgentBench: Evaluating LLMs as Agents cites Language Models can Solve Computer Tasks
- Can Dependencies Induced by LLM-Agent Workflows Be Trusted? cites AgentBench: Evaluating LLMs as Agents
- Multimodal Web Navigation with Instruction-Finetuned Foundation Models cites Language Models can Solve Computer Tasks
Cited by top-tier papers16
- MIP against Agent: Malicious Image Patches Hijacking Multimodal OS AgentsLukas Aichberger, Alasdair Paren, Guohao Li, Philip H. S. Torr et al.NeurIPS 2025 · 12 citations
- PolySkill: Learning Generalizable Skills Through Polymorphic Abstraction For Continual LearningSimon Yu, Gang Li, Weiyan Shi, Peng QiICLR 2026 · 11 citations
- UINavBench: A Framework for Comprehensive Evaluation of Interactive Digital AgentsHarsh Agrawal, Eldon Schoop, Xinlei Pan, Anuj Mahajan et al.ICCV 2025 · 9 citations
- R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement LearningYongchao Chen, Yueying Liu, Junwei Zhou, Yilun Hao et al.ICLR 2026 · 6 citations
- Can Dependencies Induced by LLM-Agent Workflows Be Trusted?Yu Yao, Yiliao Song, Yian Xie, Mengdan Fan et al.NeurIPS 2025 · 4 citations
Builds on29
- AgentBench: Evaluating LLMs as AgentsXiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu et al.ICLR 2024 · 748 citations
- GAIA: a benchmark for General AI AssistantsGrégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun et al.ICLR 2024 · 716 citations
- Language Models can Solve Computer TasksGeunwoo Kim, Pierre Baldi, Stephen McAleerNeurIPS 2023 · 539 citations
- Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent CollaborationJunyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang et al.NeurIPS 2024 · 245 citations
- Multimodal Web Navigation with Instruction-Finetuned Foundation ModelsHiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo et al.ICLR 2024 · 160 citations
Related papers
- ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon TasksYuanyi Song, Heyuan Huang, Qiqiang Lin, Yin Zhao et al.WWW 2026 · 5 citations
- Spa-Bench: a comprehensive Benchmark for Smartphone Agent EvaluationJingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang et al.ICLR 2025
- MultiAgentBench : Evaluating the Collaboration and Competition of LLM agentsKunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang et al.ACL 2025 · 97 citations
- X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic SystemPeng Wang, Ruihan Tao, Qiguang Chen, Mengkang Hu et al.ACL Findings 2025 · 5 citations
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied AgentsRui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao et al.ICML 2025
