To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis

Recent research has highlighted the importance of dataset size in scaling language models. However, large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is likely to be approaching its scaling limit for LLMs. To further enhance LLMs, a straightforward approach is to repeat the pre-training data for additional epochs. In this study, we empirically investigate three key aspects under this approach. First, we explore the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting, leading to multi-epoch degradation. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives, while less influential factors consist of dataset quality and model FLOPs. Finally, we explore whether widely used regularization can alleviate multi-epoch degradation. Most regularization techniques do not yield significant improvements, except for dropout, which demonstrates remarkable effectiveness but requires careful tuning when scaling up the model size. Additionally, we discover that leveraging mixture-of-experts (MoE) enables cost-effective and efficient hyper-parameter tuning for computationally intensive dense LLMs with comparable trainable parameters, potentially impacting efficient LLM development on a broader scale. Dataset size is more important than we thought in LLM scaling. Recent work [8] found that the pre-training dataset size plays a more significant role than previously thought and proposed compute-optimal scaling (i.e., Chinchilla scaling law), where model size and training dataset size should be scaled equally for optimal performance given a fixed computation budget. For instance, an under-trained larger model like Gopher-280B [20] can be outperformed by a well-trained smaller model like Chinchilla-70B if not enough data is used in larger model training. The intuition here is that the decreased model size can be compensated by the increased size of data. The effectiveness of the Chinchilla scaling law is further validated by the recent success of LLaMA-65B [31] and PaLM-2 [1].