ACL Findings2024
Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction
Tingchen Fu, Deng Cai, Lemao Liu, Shuming Shi, Rui Yan
Abstract
Supervised fine-tuning (SFT) on instructionfollowing corpus is a crucial approach toward the alignment of large language models (LLMs). However, the performance of LLMs on standard knowledge and reasoning benchmarks tends to suffer from deterioration at the latter stage of the SFT process, echoing the phenomenon of alignment tax. Through our pilot study, we put a hypothesis that the data biases are probably one cause behind the phenomenon. To address the issue, we introduce a simple disperse-then-merge framework. To be concrete, we disperse the instructionfollowing data into portions and train multiple sub-models using different data portions. Then we merge multiple models into a single one via model merging techniques. Despite its simplicity, our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization cites Measuring Massive Multitask Language Understanding
- FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization cites Training language models to follow instructions with human feedback
- EpiCaR: Knowing What You Don't Know Matters for Better Reasoning in LLMs cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- EpiCaR: Knowing What You Don't Know Matters for Better Reasoning in LLMs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence cites LoRA: Low-Rank Adaptation of Large Language Models
- Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence cites Measuring Massive Multitask Language Understanding
- Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- On the Loss of Context Awareness in General Instruction Fine-tuning cites Training language models to follow instructions with human feedback
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
- Learning Rewrite-Invariant Reasoning with Targeted Alternation Training cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Learning Rewrite-Invariant Reasoning with Targeted Alternation Training cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers5
- On the Loss of Context Awareness in General Instruction Fine-tuningYihan Wang, Andrew Bai, Nanyun Peng, Cho-Jui HsiehNeurIPS 2025 · 11 citations
- FlipGuard: Defending Preference Alignment against Update Regression with Constrained OptimizationMingye Zhu, Yi Liu, Quan Wang, Junbo Guo et al.EMNLP 2024 · 2 citations
- EpiCaR: Knowing What You Don't Know Matters for Better Reasoning in LLMsJe Won Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park et al.ACL 2026 · 1 citation
- Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm IntelligenceShangbin Feng, Zifeng Wang, Yike Wang, Sayna Ebrahimi et al.ICML 2025
- Learning Rewrite-Invariant Reasoning with Targeted Alternation TrainingMousa Arraf, Ido Guy, Kira RadinskyICML 2026
Builds on32
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
Related papers
- Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMsZixuan Ren, Jinliang Lu, Junhong Wu, Yang Zhao et al.ICLR 2026 · 2 citations
- PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMsZijing Wang, Yongkang Liu, Mingyang Wang, Ercong Nie et al.ACL Findings 2026
- Cure-SFT: Diagnostic-Guided Data Curation for Instruction TuningYuankang Fu, Xinrong Gong, Chen Gong, Tong Zhang et al.ICML 2026
- The Emperor's New Reasoning: Format Imitation Overshadows Genuine Mathematical Understanding in SFTLinyao Yang, Jian-Tao Huang, Yafei Lu, Zhenhui Jessie Li et al.EMNLP 2025
- Navigating the Pareto Frontier of Alignment: Spectrum-Adaptive Fine-Tuning for LLMsYaoyou Fan, Chao Zhang, Xiaoyu Tan, Chenxing Sun et al.ICML 2026
