Lune

ICLR2025Top-tier venue

Improving Pretraining Data Using Perplexity Correlations

Tristan Thrush, Christopher Potts, Tatsunori Hashimoto

Abstract

Quality pretraining data is often seen as the key to high-performance language models. However, progress in understanding pretraining data has been slow due to the costly pretraining runs required for data selection experiments. We present a framework that avoids these costs and selects high-quality pretraining data without any LLM training of our own. Our work is based on a simple observation: LLM losses on many pretraining texts are correlated with downstream benchmark performance, and selecting high-correlation documents is an effective pretraining data selection method. We build a new statistical framework for data selection centered around estimates of perplexity-benchmark correlations and perform data selection using a sample of 90 LLMs taken from the Open LLM Leaderboard on texts from tens of thousands of web domains. In controlled pretraining experiments at the 160M parameter scale on 8 benchmarks, our approach outperforms DSIR on every benchmark, while matching the best data selector found in DataComp-LM, a hand-engineered bigram classifier. We have now also updated this paper to include results from preregistered experiments with new pretraining data on an aggregation of 22 benchmarks up to the 1.4B scale, showing increasing improvements of our method over others with more scale. A pip package with full documentation can be found here: https://github.com/TristanThrush/perplexity-correlations.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

25 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining cites Data Selection for Language Models via Importance Resampling
  • Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Pre-training under infinite compute cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Pre-training under infinite compute cites Energy-Based Transformers are Scalable Learners and Thinkers
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers25

  1. Universal Model Routing for Efficient LLM InferenceWittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja et al.ICLR 2026 · 99 citations
  2. Tool Learning in the Wild: Empowering Language Models as Automatic Tool AgentsZhengliang Shi, Shen Gao, Lingyong Yan, Yue Feng et al.WWW 2025 · 59 citations
  3. Energy-Based Transformers are Scalable Learners and ThinkersAlexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han et al.ICLR 2026 · 38 citations
  4. Pre-training under infinite computeKonwoo Kim, Suhas Kotha, Percy Liang, Tatsunori HashimotoICLR 2026 · 25 citations
  5. Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian FrameworkThomson Yen, Andrew Siah, Haozhe Chen, C. Guetta et al.NeurIPS 2025 · 12 citations

Builds on7

  1. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  2. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph CompilationJason Ansel, Edward Z. Yang, Horace He, Natalia Gimelshein et al.ASPLOS 2024 · 693 citations
  3. Hyena Hierarchy: Towards Larger Convolutional Language ModelsMichael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu et al.ICML 2023 · 481 citations
  4. DoReMi: Optimizing Data Mixtures Speeds Up Language Model PretrainingSang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du et al.NeurIPS 2023 · 457 citations
  5. Data Selection for Language Models via Importance ResamplingSang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy LiangNeurIPS 2023 · 383 citations

Related papers

  1. Predictive Data Selection: The Data That Predicts Is the Data That TeachesKaShun Shum, Yuzhen Huang, Hongjian Zou, Qi Ding et al.ICML 2025
  2. DataMan: Data Manager for Pre-training Large Language ModelsRu Peng, Kexin Yang, Yawen Zeng, Junyang Lin et al.ICLR 2025
  3. Deciphering the Impact of Pretraining Data on Large Language Models through Machine UnlearningYang Zhao, Li Du, Xiao Ding, Kai Xiong et al.ACL Findings 2024 · 5 citations
  4. Scalable Data Ablation Approximations for Language Models through Modular Training and MergingClara Na, Ian Magnusson, Ananya Harsh Jha, Tom Sherborne et al.EMNLP 2024 · 2 citations
  5. Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference ModelsZachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion et al.ICLR 2025 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines