Lune

ACL Findings2025

MMInA: Benchmarking Multihop Multimodal Internet Agents

Shulin Tian, Ziniu Zhang, Liangyu Chen, Ziwei Liu

Abstract

Autonomous embodied agents live on an Internet of multimedia websites. Can they hop around multimodal websites to complete complex user tasks? Existing benchmarks fail to assess them in a realistic, evolving environment for their embodiment across websites. To answer this question, we present MMInA, a multihop and multimodal benchmark to evaluate the embodied agents for compositional Internet tasks, with several appealing properties: 1) Evolving real-world multimodal websites. Our benchmark uniquely operates on evolving real-world websites, ensuring a high degree of realism and applicability to natural user tasks. Our data includes 1,050 human-written tasks covering various domains such as shopping and travel, with each task requiring the agent to extract multimodal information from web pages as observations autonomously; 2) Multihop web browsing. Our dataset features naturally compositional tasks that require information from or actions on multiple websites to solve, to assess long-range reasoning capabilities on web tasks; 3) Holistic evaluation. We propose a novel protocol for evaluating an agent's progress in completing multihop tasks. We experiment with both standalone (multimodal) language models and heuristic-based web agents. Extensive experiments demonstrate that while long-chain multihop web tasks are easy for humans, they remain challenging for state-of-the-art web agents. We identify that agents are more likely to fail on the early hops when solving tasks with more hops, which results in lower task success rates. To address this issue, we propose a simple memory augmentation approach that replays past action trajectories to reflect. Our method significantly improves the performance of both the single-hop and multihop web browsing abilities. Our code and data are available at github.com/shulin16/MMInA.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

10 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Zero-Shot Text-to-Image Generation
  • Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Zero-Shot Text-to-Image Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Finetuned Language Models are Zero-Shot Learners
  • AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping cites Large Language Models Empowered Personalized Web Agents
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers10

  1. Large Language Models Empowered Personalized Web AgentsHongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu et al.WWW 2025 · 62 citations
  2. macOSWorld: A Multilingual Interactive Benchmark for GUI AgentsPei Yang, Hai Ci, Mike Zheng ShouNeurIPS 2025 · 34 citations
  3. Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative ModelsFan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao et al.ACL 2025 · 32 citations
  4. AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin et al.EMNLP 2024 · 5 citations
  5. AgenticShop: Benchmarking Agentic Product Curation for Personalized Web ShoppingSunghwan Kim, Ryang Heo, Yongsik Seo, Jinyoung Yeo et al.WWW 2026 · 3 citations

Builds on23

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  4. Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
  5. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations

Related papers

  1. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web TasksJing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur et al.ACL 2024 · 25 citations
  2. Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal AgentsTianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen et al.ACL 2025 · 13 citations
  3. EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied AgentsRui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao et al.ICML 2025
  4. MMhops-R1: Multimodal Multi-hop ReasoningTao Zhang, Ziqi Zhang, Zongyang Ma, Yuxin Chen et al.AAAI 2026
  5. WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code GenerationRabiul Awal, Mahsa Massoud, Aarash Feizi, Zichao Li et al.EMNLP 2025
Dusk over the sea between two cliffs drawn in fine vertical lines