EMNLP Findings2024
Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach
Dongyue Li, Ziniu Zhang, Lu Wang, Hongyang Zhang
Abstract
We study the problem of fine-tuning a language model (LM) for a target task by optimally using the information from n auxiliary tasks. This problem has broad applications in NLP, such as targeted instruction tuning and data selection in chain-of-thought fine-tuning. The key challenge of this problem is that not all auxiliary tasks are useful to improve the performance of the target task. Thus, choosing the right subset of auxiliary tasks is crucial. Conventional subset selection methods, such as forward & backward selection, are unsuitable for LM finetuning because they require repeated training on subsets of auxiliary tasks. This paper introduces a new algorithm to estimate model finetuning performances without repeated training. Our algorithm first performs multitask training using the data of all the tasks to obtain a meta initialization. Then, we approximate the model fine-tuning loss of a subset using functional values and gradients from the meta initialization. Empirically, we find that this gradient-based approximation holds with remarkable accuracy for twelve transformer-based LMs. Thus, we can now estimate fine-tuning performances on CPUs within a few seconds. We conduct extensive experiments to validate our approach, delivering a speedup of 30× over conventional subset selection while incurring only 1% error of the true fine-tuning performances. In downstream evaluations of instruction tuning and chain-of-thought fine-tuning, our approach improves over prior methods that utilize gradient or representation similarity for subset selection by up to 3.8%.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry cites LoRA: Low-Rank Adaptation of Large Language Models
- GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry cites LESS: Selecting Influential Data for Targeted Instruction Tuning
- Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation cites Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets
- QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
- Efficient Estimation of Kernel Surrogate Models for Task Attribution cites Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation
- Efficient Estimation of Kernel Surrogate Models for Task Attribution cites Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation
- Efficient Estimation of Kernel Surrogate Models for Task Attribution cites Sharpness-aware Minimization for Efficiently Improving Generalization
- Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets cites LoRA: Low-Rank Adaptation of Large Language Models
- Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets cites QLoRA: Efficient Finetuning of Quantized LLMs
- Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets cites LESS: Selecting Influential Data for Targeted Instruction Tuning
- Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation cites Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation
- Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation cites Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets
Cited by top-tier papers8
- Efficient Ensemble for Fine-tuning Language Models on Multiple DatasetsDongyue Li, Ziniu Zhang, Lu Wang, Hongyang R. ZhangACL 2025 · 8 citations
- Efficient Estimation of Kernel Surrogate Models for Task AttributionZhenshuo Zhang, Minxuan Duan, Hongyang R. ZhangICLR 2026 · 6 citations
- GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization GeometryGuanghui Min, Tianhao Huang, Ke Wan, Chen ChenICML 2026 · 3 citations
- Scalable Multi-Objective and Meta Reinforcement Learning via Gradient EstimationZhenshuo Zhang, Minxuan Duan, Youran Ye, Hongyang R. ZhangAAAI 2026 · 3 citations
- Linear-Time Demonstration Selection for In-Context Learning via Gradient EstimationZiniu Zhang, Zhenshuo Zhang, Dongyue Li, Lu Wang et al.EMNLP 2025
Builds on22
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- Sharpness-aware Minimization for Efficiently Improving GeneralizationPierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam NeyshaburICLR 2021 · 1,861 citations
- LESS: Selecting Influential Data for Targeted Instruction TuningMengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora et al.ICML 2024 · 460 citations
Related papers
- A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn’t)Nihal Nayak, Paula Rodriguez-Diaz, Neha Hulkund, Sara Beery et al.ICML 2026 · 2 citations
- GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer NetworksWeicheng Ma, Renze Lou, Kai Zhang, Lili Wang et al.EMNLP 2021 · 4 citations
- TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction TuningManish Nagaraj, Sakshi Choudhary, Utkarsh Saxena, Deepak Ravikumar et al.ICML 2026 · 2 citations
- NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation MetricJingtan Wang, Xiaoqiang Lin, Rui Qiao, Pang Wei Koh et al.ICML 2025
- Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-TuningMing Li, Yong Zhang, Shwai He, Zhitao Li et al.ACL 2024 · 16 citations
