Resonance in Weight Space: Covariate Shift Can Drive Divergence of SGD with Momentum
Kirby Banman, Liam Peet-Pare, Nidhi Hegde, Alona Fyshe, Martha White
Abstract
Most convergence guarantees for stochastic gradient descent with momentum (SGDm) rely on iid sampling. Yet, SGDm is often used outside this regime, in settings with temporally correlated input samples such as continual learning and reinforcement learning. Existing work has shown that SGDm with a decaying step-size can converge under Markovian temporal correlation. In this work, we show that SGDm under covariate shift with a fixed step-size can be unstable and diverge. In particular, we show SGDm under covariate shift is a parametric oscillator, and so can suffer from a phenomenon known as resonance. We approximate the learning system as a time varying system of ordinary differential equations, and leverage existing theory to characterize the system's divergence/convergence as resonant/nonresonant modes. The theoretical result is limited to the linear setting with periodic covariate shift, so we empirically supplement this result to show that resonance phenomena persist even under non-periodic covariate shift, nonlinear dynamics with neural networks, and optimizers other than SGDm.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on4
- The Non-IID Data Quagmire of Decentralized Machine LearningKevin Hsieh, Amar Phanishayee, Onur Mutlu, Phillip B. GibbonsICML 2020 · 672 citations
- Least Squares Regression with Markovian Data: Fundamental Limits and AlgorithmsDheeraj Nagaraj, Xian Wu, Guy Bresler, Prateek Jain et al.NeurIPS 2020 · 73 citations
- Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient NoiseUmut Simsekli, Lingjiong Zhu, Yee Whye Teh, Mert GürbüzbalabanICML 2020 · 58 citations
- Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian SamplingHuaqing Xiong, Tengyu Xu, Yingbin Liang, Wei ZhangAAAI 2021 · 37 citations
Related papers
- Noise and Fluctuation of Finite Learning Rate Stochastic Gradient DescentKangqiao Liu, Liu Ziyin, Masahito UedaICML 2021 · 46 citations
- Random Scaling and Momentum for Non-smooth Non-convex OptimizationQinzi Zhang, Ashok CutkoskyICML 2024 · 10 citations
- On the Convergence Direction of Gradient DescentShuo Chen, Xiaolong Li, Jiaying Peng, Yao ZhaoICLR 2026
- On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic OptimizationSharan Sahu, Cameron Hogan, Martin WellsICML 2026 · 1 citation
- An Improved Analysis of Stochastic Gradient Descent with MomentumYanli Liu, Yuan Gao, Wotao YinNeurIPS 2020 · 328 citations
