ACL Findings2025
StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7, 000+ Real-World APIs
Zhicheng Guo, Sijie Cheng, Yuchen Niu, Hao Wang, Sicheng Zhou, Wenbing Huang, Yang Liu
Abstract
The rapid advancement of large language models (LLMs) has spurred significant interest in tool learning, where LLMs are augmented with external tools to tackle complex tasks. However, existing tool environments face challenges in balancing stability, scalability, and realness, particularly for benchmarking purposes. To address this problem, we propose MirrorAPI, a novel framework that trains specialized LLMs to accurately simulate real API responses, effectively acting as "mirrors" to tool environments. Using a comprehensive dataset of request-response pairs from 7,000+ APIs, we employ supervised fine-tuning and chain-ofthought reasoning to enhance simulation fidelity. MirrorAPI achieves superior accuracy and stability compared to state-of-the-art methods, as demonstrated by its performance on the newly constructed MirrorAPI-Bench and its integration into StableToolBench.
Ask Lune
Ask anything. The answer quotes the paper.
Lune reads all of it and the papers around it, then shows you the passages behind every answer.
Builds on20
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Gorilla: Large Language Model Connected with Massive APIsShishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. GonzalezNeurIPS 2024 · 1,715 citations
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language AgentsShunyu Yao, Howard Chen, John Yang, Karthik NarasimhanNeurIPS 2022 · 1,477 citations
- WebArena: A Realistic Web Environment for Building Autonomous AgentsShuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou et al.ICLR 2024 · 1,197 citations
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive CritiquingZhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen et al.ICLR 2024 · 699 citations
Related papers
- StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language ModelsZhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang et al.ACL Findings 2024 · 22 citations
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIsYujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu et al.ICLR 2024 · 1,469 citations
- Learning Evolving Tools for Large Language ModelsGuoxin Chen, Zhong Zhang, Xin Cong, Fangda Guo et al.ICLR 2025
- Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection LearningZhiyuan Ma, Jiayu Liu, Xianzhen Luo, Zhenya Huang et al.KDD 2025 · 4 citations
- Do LLMs Catch Their Own Mistakes? A Comprehensive Benchmark for Reflective Tool Use LLMsZheyuan Liu, Liqiang Xiao, Yang Li, Hyokun Yun et al.ACL Findings 2026
