Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning

Performance evaluation of the Holistic Data Scheduler (HDS). All results are from a Pythia-1B model trained for 50 billion tokens on The Pile dataset. (a) Unweighted average validation perplexity across the 22 domains of The Pile. (b) 0-shot accuracy on the MMLU benchmark.