Emergence and scaling laws in SGD learning of shallow neural networks

We study the complexity of online stochastic gradient descent (SGD) for learning a two-layer neural network with 𝑃 neurons on isotropic Gaussian data:

where the activation 𝜎 is an even function with information exponent 𝑘 * > 2 (defined as the lowest degree in Hermite expansion), v * 𝑝 𝑝∈ [ 𝑃] ⊂ R 𝑑 are orthonormal signal directions, and non-negative second-layer coefficients satisfy 𝑝 𝑎 2 𝑝 = 1. We focus on the challenging "extensive-width" regime 𝑃 ≫ 1 and permit diverging condition number in the second-layer, covering as a special case the power-law scaling 𝑎 𝑝 ≍ 𝑝 -𝛽 where 𝛽 ∈ R ≥0 . We provide a precise analysis of SGD dynamics for the training of a student two-layer network to minimize the mean squared error (MSE) objective, and identify sharp transition times to recover each signal direction. In the power-law setting, we characterize scaling law exponents for the MSE loss with respect to the number of training samples and SGD steps, as well as the number of trainable parameters. Our analysis entails that while the learning of individual teacher neurons exhibits abrupt transitions, the juxtaposition of 𝑃 ≫ 1 emergent learning curves at different timescales leads to a smooth scaling law in the cumulative objective.

Remark. We focus on high information exponent IE(𝜎) > 2 link functions as in [OSSW24, SBH24, GWB25]. This setting entails that the learning of each single-index task is "hard" in the sense that online SGD exhibits a long loss plateau, and we utilize this assumption to prove (approximate) decoupling of individual tasks. The condition on even 𝜎 simplifies the analysis by removing the 1/2 probability of neurons initialized in the wrong hemisphere (see e.g., [BAGJ21]).