ACL Findings2025
Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning
Ziang Ye, Zhenru Zhang, Yang Zhang, Jianxin Ma, Junyang Lin, Fuli Feng
Abstract
When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles-specifically, reasoning tokens versus boilerplate tokens (e.g., those governing output format)-differ significantly in importance and learning complexity, necessitating their disentanglement and distinct treatment. To address this, we propose a novel Shuffle-Aware Discriminator (SHAD) for adaptive token discrimination. SHAD classifies tokens by exploiting predictability differences observed after shuffling input-output combinations across samples: boilerplate tokens, due to their repetitive nature among samples, maintain predictability, whereas reasoning tokens do not. Using SHAD, we propose the Reasoning-highlighted Fine-Tuning (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning, yielding notable performance gains over common Supervised Fine-Tuning (SFT).
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Gorilla: Large Language Model Connected with Massive APIs cites Toolformer: Language Models Can Teach Themselves to Use Tools
- Gorilla: Large Language Model Connected with Massive APIs cites HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
- Gorilla: Large Language Model Connected with Massive APIs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Reflexion: language agents with verbal reinforcement learning cites Toolformer: Language Models Can Teach Themselves to Use Tools
- Reflexion: language agents with verbal reinforcement learning cites HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
- Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face cites Toolformer: Language Models Can Teach Themselves to Use Tools
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds on17
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceYongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li et al.NeurIPS 2023 · 1,778 citations
- Gorilla: Large Language Model Connected with Massive APIsShishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. GonzalezNeurIPS 2024 · 1,715 citations
Related papers
- Incentivizing LLM Reasoning via Reinforcement Learning with Functional Monte Carlo Tree SearchKongcheng Zhang, QI YAO, Baisheng Lai, Jiaxing Huang et al.ICLR 2026
- Explainable Token-level Noise Filtering for LLM Fine-tuning DatasetsYuchen Yang, Wenze Lin, Enhao Huang, Zhixuan Chu et al.ICLR 2026 · 1 citation
- Tailored Primitive Initialization is the Secret Key to Reinforcement LearningYihang Yao, Guangtao Zeng, Raina Wu, Yang Zhang et al.ACL 2026 · 1 citation
- Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMsHaoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma et al.ICLR 2026 · 24 citations
- Don't Force the Fit: Bounded Log-Likelihood Loss for Enhanced Reasoning in Large Language ModelsFeng Zhao, Hong Zhang, Yu Yang, Ruilin Zhao et al.ICML 2026
