LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su
Abstract
Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM's 'imagination' to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on Tool-Bench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
14 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery cites Toolformer: Language Models Can Teach Themselves to Use Tools
- AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation cites Reflexion: language agents with verbal reinforcement learning
- AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation cites Toolformer: Language Models Can Teach Themselves to Use Tools
- Reflexion: language agents with verbal reinforcement learning cites Self-Refine: Iterative Refinement with Self-Feedback
- Reflexion: language agents with verbal reinforcement learning cites Toolformer: Language Models Can Teach Themselves to Use Tools
- CITI: Enhancing Tool Utilizing Ability in Large Language Models Without Sacrificing General Performance cites Toolformer: Language Models Can Teach Themselves to Use Tools
- GTool: Graph Enhanced Tool Planning with Large Language Model cites Toolformer: Language Models Can Teach Themselves to Use Tools
- Toolformer: Language Models Can Teach Themselves to Use Tools cites Retrieval Augmented Language Model Pre-Training
Cited by top-tier papers14
- Improving Code Localization with Repository MemoryBoshi Wang, Weijian Xu, Yunsheng Li, Xuemei Gao et al.ICLR 2026 · 20 citations
- GTool: Graph Enhanced Tool Planning with Large Language ModelWenjie Chen, Di Yao, Wenbin Li, Xuying Meng et al.ICLR 2026 · 8 citations
- ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific DiscoveryZiru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang et al.ICLR 2025 · 6 citations
- AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task GenerationMengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun et al.KDD 2025 · 6 citations
- CITI: Enhancing Tool Utilizing Ability in Large Language Models Without Sacrificing General PerformanceYupu Hao, Pengfei Cao, Zhuoran Jin, Huanxuan Liao et al.AAAI 2025 · 6 citations
Builds on31
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
Related papers
- StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7, 000+ Real-World APIsZhicheng Guo, Sijie Cheng, Yuchen Niu, Hao Wang et al.ACL Findings 2025 · 1 citation
- Tool Learning in the Wild: Empowering Language Models as Automatic Tool AgentsZhengliang Shi, Shen Gao, Lingyong Yan, Yue Feng et al.WWW 2025 · 59 citations
- Tool-Planner: Task Planning with Clusters across Multiple ToolsYanming Liu, Xinyue Peng, Jiannan Cao, Shi Bo et al.ICLR 2025
- ToolExpNet: Optimizing Multi-Tool Selection in LLMs with Similarity and Dependency-Aware Experience NetworksZijing Zhang, Zhanpeng Chen, He Zhu, Ziyang Chen et al.ACL Findings 2025
- Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection LearningZhiyuan Ma, Jiayu Liu, Xianzhen Luo, Zhenya Huang et al.KDD 2025 · 4 citations
