Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks

Deeper modern architectures are costly to tune, and the base learning rate is often one of the most sensitive hyperparameters. Maximal Update Parametrization ($\mu$P) helps explain why many hyperparameters transfer across width. Yet depthwise learning-rate scaling is less understood for modern architectures with convolution, residual aggregation, and attention. To unify various non-recurrent multi-path neural networks such as CNNs, ResNets, and Transformers, we introduce an architecture-dependent notion of effective depth. Under stabilizing initializations and a maximal-update criterion, we derive a shared leading-order -3/2 law for the base learning-rate scale as effective depth grows. Here, the budget controls typical one-step representation-update energy at initialization, and effective depth counts sequential update-bearing units while absorbing fixed local structure into constants. Experiments across diverse architectures confirm the predicted slope and enable reliable zero-shot transfer of learning rates across depths and widths, turning depth scaling into a predictable hyperparameter-transfer problem.