Scaling Laws for Code: Every Programming Language Matters

Large language models (LLMs) are powerful but costly to train, with scaling laws predicting performance from model size, data, and compute. However, different programming languages (PLs) have varying impacts during pretraining that significantly affect base model performance, leading to inaccurate performance prediction. Existing works focus on languageagnostic settings, neglecting the inherently multilingual nature of modern software development. Therefore, it is first necessary to investigate the scaling laws of different PLs, and then consider their mutual influences to arrive at the final multilingual scaling law. In this paper, we present the first systematic exploration of scaling laws for multilingual code pre-training, conducting over 1000+ experiments (Equivalent to 336,000+ H800 hours) across multiple PLs, model sizes (0.2B to 14B parameters), and dataset sizes (1T tokens). We establish scaling laws for code LLMs, showing interpreted languages benefit more from scale than compiled ones. Multilingual pre-training provides synergistic benefits between similar languages, with parallel pairing enhancing cross-lingual abilities. We propose a proportion-dependent scaling law that optimally allocates training tokens by prioritizing high-utility languages (e.g., Python), balancing high-synergy pairs (e.g., JavaScript-TypeScript), and reducing fastsaturating languages (Rust), achieving superior performance versus uniform distribution.