ACL Findings2024
Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning
Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Zhouhao Sun, Shi Jun, Ting Liu, Bing Qin
Abstract
Through pretraining on a corpus with various sources, Large Language Models (LLMs) have gained impressive performance. However, the impact of each component of the pretraining corpus remains opaque. As a result, the organization of the pretraining corpus is still empirical and may deviate from the optimal. To address this issue, we systematically analyze the impact of 48 datasets from 5 major categories of pretraining data of LLMs and measure their impacts on LLMs using benchmarks about nine major categories of model capabilities. Our analyses provide empirical results about the contribution of multiple corpora on the performances of LLMs, along with their joint impact patterns, including complementary, orthogonal, and correlational relationships. We also identify a set of "high-impact data" such as Books that is significantly related to a set of model capabilities. These findings provide insights into the organization of data to support more efficient pretraining of LLMs.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers4
- UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data SelectionYang Zhao, Kai Xiong, Xiao Ding, Li Du et al.NeurIPS 2025 · 4 citations
- Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data SelectionYang Zhao, Li Du, Xiao Ding, Yangou Ouyang et al.ACL 2025 · 3 citations
- Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient ConcentrationYang Zhao, Yangou Ouyang, Xiao Ding, Hepeng Wang et al.ACL 2026 · 2 citations
- TransLibEval: Demystify Large Language Models' Capability in Third-Party Library-Targeted Code TranslationPengyu Xue, Kunwu Zheng, Zhen Yang, Yifei Pei et al.FSE 2026 · 1 citation
Builds on11
- Estimating Training Data Influence by Tracing Gradient DescentGarima Pruthi, Frederick Liu, Satyen Kale, Mukund SundararajanNeurIPS 2020 · 784 citations
- Remember What You Want to Forget: Algorithms for Machine UnlearningAyush Sekhari, Jayadev Acharya, Gautam Kamath, Ananda Theertha SureshNeurIPS 2021 · 516 citations
- Amnesiac Machine LearningLaura Graves, Vineel Nagisetty, Vijay GaneshAAAI 2021 · 416 citations
- Deduplicating Training Data Mitigates Privacy Risks in Language ModelsNikhil Kandpal, Eric Wallace, Colin RaffelICML 2022 · 395 citations
- QASC: A Dataset for Question Answering via Sentence CompositionTushar Khot, Peter Clark, Michal Guerquin, Peter Jansen et al.AAAI 2020 · 387 citations
Related papers
- Common Corpus: The Largest Collection of Ethical Data for LLM Pre-TrainingPierre-Carl Langlais, Pavel Chizhov, Catherine Arnett, Carlos Rosas Hinostroza et al.ICLR 2026 · 22 citations
- Improving Pretraining Data Using Perplexity CorrelationsTristan Thrush, Christopher Potts, Tatsunori HashimotoICLR 2025
- Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining ResearchLuca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk et al.ACL 2024
- Scalable Data Ablation Approximations for Language Models through Modular Training and MergingClara Na, Ian Magnusson, Ananya Harsh Jha, Tom Sherborne et al.EMNLP 2024 · 2 citations
- Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and PitfallsFeiyang Kang, Newsha Ardalani, Michael Kuchnik, Youssef Emad et al.EMNLP 2025
