EMNLP Findings2025
Rethinking Data Selection at Scale: Random Selection is Almost All You Need
Tingyu Xia, Bowen Yu, Kai Dang, An Yang, Yuan Wu, Yuan Tian, Yi Chang, Junyang Lin
Abstract
Supervised fine-tuning (SFT) is crucial for aligning Large Language Models (LLMs) with human instructions. The primary goal during SFT is to select a small yet representative subset of training data from the larger pool, such that fine-tuning with this subset achieves results comparable to or even exceeding those obtained using the entire dataset. However, most existing data selection techniques are designed for small-scale data pools, which fail to meet the demands of real-world SFT scenarios. In this paper, we replicated several self-scoring methods-those that do not rely on external model assistance-on two millionscale datasets, and found that nearly all methods struggled to significantly outperform random selection when dealing with such largescale data pools. Moreover, our comparisons suggest that, during SFT, diversity in data selection is more critical than simply focusing on high-quality data. We also analyzed the limitations of several current approaches, explaining why they perform poorly on largescale datasets and why they are unsuitable for such contexts. Finally, we found that filtering data by token length offers a stable and efficient method for improving results. This approach, particularly when training on longtext data, proves highly beneficial for relatively weaker base models, such as Llama3. The code is available at https://github.com/ xiatingyu/SFT-DataSelection-at-scale.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training cites Measuring Massive Multitask Language Understanding
- SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training cites LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning
- SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training cites Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions cites Measuring Massive Multitask Language Understanding
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions cites CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
- Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations cites Measuring Massive Multitask Language Understanding
- Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
- LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning cites Measuring Massive Multitask Language Understanding
- Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric cites WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions
Cited by top-tier papers8
- LEAD: Iterative Data Selection for Efficient LLM Instruction TuningXiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas et al.VLDB 2026 · 16 citations
- Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable MetricYuming Yang, Yang Nan, Junjie Ye, Shihan Dou et al.ACL 2025 · 15 citations
- Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal ActivationsDa Ma, Gonghu Shang, Zhi Chen, Libo Qin et al.NeurIPS 2025 · 6 citations
- SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model TrainingPowei Chang, Jinpeng Zhang, Bowen Chen, Chenyu Wang et al.ICLR 2026 · 5 citations
- Computational Budget Should Be Considered in Data SelectionWeilin Wan, Weizhong Zhang, Cheng JinNeurIPS 2025 · 2 citations
Builds on21
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model SocietyGuohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin et al.NeurIPS 2023 · 1,975 citations
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsCan Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng et al.ICLR 2024 · 1,206 citations
Related papers
- How to Train Long-Context Language Models (Effectively)Tianyu Gao, Alexander Wettig, Howard Yen, Danqi ChenACL 2025
- ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuningXiaohan Qin, Victor Wang, Ning Liao, Cancheng Zhang et al.ICLR 2026 · 3 citations
- Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment QualityYuto Harada, Yusuke Yamauchi, Yusuke Oda, Yohei Oseki et al.EMNLP 2025
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-TuningHao Zhao, Maksym Andriushchenko, Francesco Croce, Nicolas FlammarionICML 2024 · 96 citations
- Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-TuningMing Li, Yong Zhang, Shwai He, Zhitao Li et al.ACL 2024 · 16 citations
