Lune

EMNLP Findings2025

TTPA: Token-level Tool-use Preference Alignment Training Framework with Fine-grained Evaluation

Chengrui Huang, Shen Gao, Zhengliang Shi, Dongsheng Wang, Shuo Shang

Abstract

Existing tool-learning methods usually rely on supervised fine-tuning, they often overlook finegrained optimization of internal tool call details, leading to limitations in preference alignment and error discrimination. To overcome these challenges, we propose Token-level Tooluse Preference Alignment Training Framework (TTPA), a training paradigm for constructing token-level tool-use preference datasets that align LLMs with fine-grained preferences using a novel error-oriented scoring mechanism. TTPA first introduces reversed dataset construction, a method for creating high-quality, multiturn tool-use datasets by reversing the generation flow. Additionally, we propose Preference Oriented Tool-use Dataset Construction to capture fine-grained preferences by modeling token-level differences during generation. To address biases in scoring, we introduce the Error-oriented Scoring Mechanism, which quantifies tool-call errors and can be used as a training signal. Extensive experiments on three diverse benchmark datasets demonstrate that TTPA significantly improves tool-using performance while showing strong generalization ability across models and datasets. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. In-the-Flow Agentic System Optimization for Effective Planning and Tool UseZhuofeng Li, Haoxiang Zhang, Seungju Han, Sheng Liu et al.ICLR 2026 · 65 citations
  2. DPEPO: Diverse Parallel Exploration Policy Optimization for LLM-based AgentsJunShuo Zhang, Chengrui Huang, Feng Guo, Zihan Li et al.ACL 2026

Builds on17

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. Gorilla: Large Language Model Connected with Massive APIsShishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. GonzalezNeurIPS 2024 · 1,715 citations
  3. Plug and Play Language Models: A Simple Approach to Controlled Text GenerationSumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung et al.ICLR 2020 · 1,166 citations
  4. AgentBench: Evaluating LLMs as AgentsXiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu et al.ICLR 2024 · 748 citations
  5. Executable Code Actions Elicit Better LLM AgentsXingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang et al.ICML 2024 · 436 citations

Related papers

  1. TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool UseJunjie Ye, Yilong Wu, Sixian Li, Yuming Yang et al.EMNLP Findings 2025
  2. Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference TreesSijia Chen, Yibo Wang, Yi-Feng Wu, Qingguo Chen et al.NeurIPS 2024 · 51 citations
  3. Selective Preference Optimization via Token-Level Reward Function EstimationKailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang et al.EMNLP 2025 · 18 citations
  4. Token-level Direct Preference OptimizationYongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang et al.ICML 2024 · 136 citations
  5. Teaching Your Models to Understand Code via Focal Preference AlignmentJie Wu, Haoling Li, Xin Zhang, Xiao Liu et al.EMNLP 2025
Dusk over the sea between two cliffs drawn in fine vertical lines