BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

Effective data selection is essential for pretraining large language models (LLMs), improving efficiency and generalization to downstream tasks. However, existing approaches often rely on external pretrained models, making it difficult to separate the benefits of data selection from those introduced by external models. In addition, many methods estimate data importance from a fixed model state or short-horizon update, making it hard to capture how data preference changes as the model evolves during pretraining. In this paper, we introduce BLISS (BileveL Influence Scoring method for data Selection), a lightweight data selection method that operates entirely from scratch, without external pretrained oracle models, while modeling dynamic data preference. BLISS uses a small proxy model as a surrogate for the LLM and trains a score model to estimate sample importance through multi-step proxy updates induced by score-weighted training data. We formulate data selection as a bilevel optimization problem: the upper-level objective optimizes the score model to assign sample weights, so minimizing the lower-level weighted training loss improves validation performance. Once optimized, the score model predicts influence scores, enabling efficient selection of high-quality samples for LLM pretraining. We validate BLISS by pretraining 410M/1B/2.8B Pythia and LLaMA-0.5B models on selected C4 subsets. Under the 1B setting, BLISS achieves a $1.7\times$ speedup in reaching the same performance as the state-of-the-art method, while delivering superior performance across multiple downstream tasks.