Lune

ACL Findings2026

Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments

Junjie Ye, Changhao Jiang, Zhengyin Du, Yufei Xu, Xuesong Yao, Zhiheng Xi, Xiaoran Fan, Qi Zhang, Tao Gui, Xuanjing Huang, Jiecao Chen

Abstract

Effective tool use is essential for large language models (LLMs) to interact with their environment. However, progress is limited by the lack of efficient reinforcement learning (RL) frameworks specifically designed for tool use, due to challenges in constructing stable training environments and designing verifiable reward mechanisms. To address this, we propose an automated environment construction pipeline, incorporating scenario decomposition, document generation, function integration, complexity scaling, and localized deployment. This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools. Additionally, we introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution. When combined with trajectory data collected from the constructed environments, this mechanism integrates seamlessly with standard RL algorithms to facilitate feedback-driven model training. Experiments on LLMs of varying scales demonstrate that our approach significantly enhances the models' tool-use performance without degrading their general capabilities. Our analysis suggests that these gains result from improved context understanding and reasoning, driven by updates to the lower-layer MLP parameters in models. Code and data are available at https://github.com/byt edance/FTRL.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training cites Toolformer: Language Models Can Teach Themselves to Use Tools
  • Scaling Agentic Capabilities via Grounded Interaction Synthesis cites ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
  • GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction cites Language Models are Few-Shot Learners
  • ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs cites Toolformer: Language Models Can Teach Themselves to Use Tools
  • ToolRL: Reward is All Tool Learning Needs cites Toolformer: Language Models Can Teach Themselves to Use Tools
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Aegis: Automated Error Generation and Attribution for Multi-Agent SystemsFanqi Kong, Ruijie Zhang, Huaxiao Yin, Guibin Zhang et al.ICLR 2026 · 16 citations
  2. Scaling Agentic Capabilities via Grounded Interaction SynthesisWenhang Shi, Jinhao Dong, Yiren Chen, Zhe Zhao et al.KDD 2026 · 1 citation
  3. ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent TrainingDunwei Tu, Hongyan Hao, Hansi Yang, Yihao Chen et al.ICML 2026
  4. AgentGym-RL: An Open-Source Framework to Train LLM Agents for Long-Horizon Decision Making via Multi-Turn RLZhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang et al.ICLR 2026

Builds on14

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
  3. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIsYujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu et al.ICLR 2024 · 1,469 citations
  4. ToolRL: Reward is All Tool Learning NeedsCheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang et al.NeurIPS 2025 · 387 citations
  5. GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructionRui Yang, Lin Song, Yanwei Li, Sijie Zhao et al.NeurIPS 2023 · 340 citations

Related papers

  1. Tool Learning in the Wild: Empowering Language Models as Automatic Tool AgentsZhengliang Shi, Shen Gao, Lingyong Yan, Yue Feng et al.WWW 2025 · 59 citations
  2. EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic SynthesisXiaoshuai Song, Haofei Chang, Guanting Dong, Yutao Zhu et al.ACL Findings 2026
  3. From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven InteractionsChangle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai et al.ICLR 2025
  4. Generalizable End-to-End Tool-Use RL with Synthetic CodeGymWeihua Du, Hailei Gong, Zhan Ling, Kang Liu et al.ICLR 2026 · 13 citations
  5. MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP ToolsWenhao Wang, Peizhi Niu, Zhao Xu, Zhaoyu Chen et al.ACL 2026 · 8 citations
Dusk over the sea between two cliffs drawn in fine vertical lines