Understanding MARS: When Scaling Momentum Provably Helps

MARS (Yuan et al., 2025) has recently emerged as a strong optimizer for large language model (LLM) training by scaling the correction term in momentum-based variance reduction (MVR). However, existing theory does not explain why this modification can improve convergence over the unscaled MVR choice $\gamma=1$. In this paper, we provide a theoretical explanation for this phenomenon. We introduce $\gamma$-similarity, a refined similarity condition that captures how the scaling coefficient interacts with the stochastic gradient-difference structure. This condition recovers standard similarity at $\gamma=1$ and smoothness at $\gamma=0$. Using $\gamma$-similarity, we derive convergence guarantees for fixed-$\gamma$ MARS whose complexity depends explicitly on $\gamma$ and the corresponding $\gamma$-similarity constant. The bound reveals why small values of $\gamma$ can be beneficial: they may reduce the similarity term enough to outweigh the penalty from deviating from MVR. We prove that optimizing $\gamma$ gives MARS a lower complexity guarantee than MVR. Experiments with MARS-AdamW on GPT-style LLM pretraining corroborate the theory, showing that properly chosen small values of $\gamma$ improve token efficiency over $\gamma=1$ and AdamW under a fixed training protocol.