ACL Findings2025
MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space
Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, Kai Chen
Abstract
Data quality and diversity are key to the construction of effective instruction-tuning datasets. With the increasing availability of open-source instruction-tuning datasets, it is advantageous to automatically select high-quality and diverse subsets from a vast amount of data. Existing methods typically prioritize instance quality and use heuristic rules to maintain diversity. However, this absence of a comprehensive view of the entire collection often leads to suboptimal results. Moreover, heuristic rules generally focus on distance or clustering within the embedding space, which fails to accurately capture the intent of complex instructions in the semantic space. To bridge this gap, we propose a unified method for quantifying the information content of datasets. This method models the semantic space by constructing a label graph and quantifies diversity based on the distribution of information within the graph. Based on such a measurement, we further introduce an efficient sampling method that selects data samples iteratively to Maximize the Information Gain (MIG) in semantic space. Experiments on various datasets and base models demonstrate that MIG consistently outperforms state-of-the-art methods. Notably, the model fine-tuned with 5% Tulu3 data sampled by MIG achieves comparable performance to the official SFT model trained on the full dataset, with improvements of +5.73% on AlpacaEval and +6.89% on Wildbench.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective cites Measuring Massive Multitask Language Understanding
- What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective cites Language Models are Few-Shot Learners
- What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective cites Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective cites AlpaGasus: Training a Better Alpaca with Fewer Data
- Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
- SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training cites Measuring Massive Multitask Language Understanding
- SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training cites Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- Instruction Data Selection via Answer Divergence cites Measuring Massive Multitask Language Understanding
- Instruction Data Selection via Answer Divergence cites Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- Instruction Data Selection via Answer Divergence cites AlpaGasus: Training a Better Alpaca with Fewer Data
- CypherSmith: Transforming Text-to-Cypher Generation for LLMs with Synthetic Data cites AlpaGasus: Training a Better Alpaca with Fewer Data
- AlpaGasus: Training a Better Alpaca with Fewer Data cites Measuring Massive Multitask Language Understanding
- AlpaGasus: Training a Better Alpaca with Fewer Data cites Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Cited by top-tier papers4
- SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model TrainingPowei Chang, Jinpeng Zhang, Bowen Chen, Chenyu Wang et al.ICLR 2026 · 5 citations
- Instruction Data Selection via Answer DivergenceBo Li, Mingda Wang, Shikun Zhang, Wei YeACL 2026 · 4 citations
- What Makes Good Instruction-Tuning Data? An In-Context Learning PerspectiveGuangzeng Han, Xiaolei HuangACL 2026 · 1 citation
- CypherSmith: Transforming Text-to-Cypher Generation for LLMs with Synthetic DataZeyu Zhang, Kexuan Sun, Zheng Tang, Jens-S. Vöckler et al.ACL 2026
Builds on14
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- OpenChat: Advancing Open-source Language Models with Mixed-Quality DataGuan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li et al.ICLR 2024 · 328 citations
- AlpaGasus: Training a Better Alpaca with Fewer DataLichang Chen, Shiyang Li, Jun Yan, Hai Wang et al.ICLR 2024 · 295 citations
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve ThemMirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann et al.ACL Findings 2023 · 253 citations
Related papers
- Data Diversity Matters for Robust Instruction TuningAlexander Bukharin, Shiyang Li, Zhengyang Wang, Jingfeng Yang et al.EMNLP Findings 2024 · 13 citations
- DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning CollectionsHaebin Shin, Lei Ji, Xiao Liu, Zhiwei Yu et al.ACL Findings 2026
- From Selection to Refinement: Iterative Optimization for Instruction DataHang Hu, Ziyan Liu, Rujie Wen, Ruihui Hou et al.ACL 2026
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction TuningWei Liu, Weihao Zeng, Keqing He, Yong Jiang et al.ICLR 2024 · 369 citations
- Priority on High-Quality: Selecting Instruction Data via Consistency Verification of Noise InjectionHong Zhang, Feng Zhao, Ruilin Zhao, Cheng Yan et al.EMNLP 2025
