API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li
Abstract
Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs' capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
109 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TravelPlanner: A Benchmark for Real-World Planning with Language Agents cites Toolformer: Language Models Can Teach Themselves to Use Tools
- TravelPlanner: A Benchmark for Real-World Planning with Language Agents cites Language Models are Few-Shot Learners
- Gorilla: Large Language Model Connected with Massive APIs cites Toolformer: Language Models Can Teach Themselves to Use Tools
- Gorilla: Large Language Model Connected with Massive APIs cites Language Models are Few-Shot Learners
- ToolRL: Reward is All Tool Learning Needs cites Gorilla: Large Language Model Connected with Massive APIs
- ToolRL: Reward is All Tool Learning Needs cites Toolformer: Language Models Can Teach Themselves to Use Tools
- Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast cites Toolformer: Language Models Can Teach Themselves to Use Tools
- Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast cites Language Models are Few-Shot Learners
- Prompt Injection Attack to Tool Selection in LLM Agents cites Gorilla: Large Language Model Connected with Massive APIs
- ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings cites Language Models are Few-Shot Learners
Cited by top-tier papers109
- ToolRL: Reward is All Tool Learning NeedsCheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang et al.NeurIPS 2025 · 387 citations
- TravelPlanner: A Benchmark for Real-World Planning with Language AgentsJian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu et al.ICML 2024 · 376 citations
- Prompt Injection Attack to Tool Selection in LLM AgentsJiawen Shi, Zenghui Yuan, Guiyao Tie, Pan Zhou et al.NDSS 2026 · 181 citations
- Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially FastXiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du et al.ICML 2024 · 128 citations
- AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent SystemsYingxuan Yang, Huacan Chai, Shuai Shao, Yuanyi Song et al.NeurIPS 2025 · 104 citations
Builds on8
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
- Gorilla: Large Language Model Connected with Massive APIsShishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. GonzalezNeurIPS 2024 · 1,715 citations
- Preference Ranking Optimization for Human AlignmentFeifan Song, Bowen Yu, Minghao Li, Haiyang Yu et al.AAAI 2024 · 357 citations
- ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool EmbeddingsShibo Hao, Tianyang Liu, Zhen Wang, Zhiting HuNeurIPS 2023 · 315 citations
Related papers
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIsYujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu et al.ICLR 2024 · 1,469 citations
- PlanningArena: A Modular Benchmark for Multidimensional Evaluation of Planning and Tool LearningZihan Zheng, Tianle Cui, Chuwen Xie, Jiahui Pan et al.ACL 2025 · 3 citations
- Meta-Tool: Unleash Open-World Function Calling Capabilities of General-Purpose Large Language ModelsShengqian Qin, Yakun Zhu, Linjie Mu, Shaoting Zhang et al.ACL 2025 · 3 citations
- Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex ScenariosShijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu et al.ACL Findings 2024 · 15 citations
- StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language ModelsZhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang et al.ACL Findings 2024 · 22 citations
