Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning

Chenhao Dang; Jing Ma; Mingjie Liao

doi:10.1145/3770854.3780325

KDD2026Top-tier venue

Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning

Chenhao Dang, Jing Ma, Mingjie Liao

arXiv DOI Publisher

2026Year

Abstract

Performance evaluation of the Holistic Data Scheduler (HDS). All results are from a Pythia-1B model trained for 50 billion tokens on The Pile dataset. (a) Unweighted average validation perplexity across the 22 domains of The Pile. (b) 0-shot accuracy on the MMLU benchmark.

Ask about this paper

Your agent reads all of it.

Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.

What it claims
How it compares
Where it falls short

You ask

Your agent calls

Luneget_paper_fulltext

Ask in Lune

Free to start. No credit card required.

Builds on10

Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
DoReMi: Optimizing Data Mixtures Speeds Up Language Model PretrainingSang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du et al.NeurIPS 2023 · 457 citations

Abstract

Ask about this paper

Your agent reads all of it.

Builds on10

Related papers