A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias
Wei-Kai Chang, Rajiv Khanna
Abstract
Understanding the dynamics of optimization in deep learning is increasingly important as models scale. While stochastic gradient descent (SGD) and its variants reliably find solutions that generalize well, the mechanisms driving this generalization remain unclear. Notably, these algorithms often prefer flatter or simpler minima-particularly in overparameterized settings. Prior work has linked flatness to generalization, and methods like Sharpness-Aware Minimization (SAM) explicitly encourage flatness, but a unified theory connecting data structure, optimization dynamics, and the nature of learned solutions is still lacking. In this work, we develop a linear stability framework that analyzes the behavior of SGD, random perturbations, and SAM-particularly in two-layer ReLU networks. Central to our analysis is a coherence measure that quantifies how gradient curvature aligns across data points, revealing why certain minima are stable and favored during training. (Code are available in: https://github.com/changwk1001/ Stability_Analysis_and_Simplicity-Bias.git)
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Certification of Machine Learning Models via Directional Sharpness cites Fantastic Generalization Measures and Where to Find Them
- Certification of Machine Learning Models via Directional Sharpness cites Sharpness-aware Minimization for Efficiently Improving Generalization
- ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks cites Fantastic Generalization Measures and Where to Find Them
- ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks cites Sharpness-aware Minimization for Efficiently Improving Generalization
- Membership Privacy Risks of Sharpness Aware Minimization cites ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
- Membership Privacy Risks of Sharpness Aware Minimization cites Sharpness-aware Minimization for Efficiently Improving Generalization
- Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization cites Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
- Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization cites Fantastic Generalization Measures and Where to Find Them
- Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization cites Membership Privacy Risks of Sharpness Aware Minimization
- Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization cites Sharpness-aware Minimization for Efficiently Improving Generalization
- Sharpness-aware Minimization for Efficiently Improving Generalization cites Fantastic Generalization Measures and Where to Find Them
Cited by top-tier papers3
- Membership Privacy Risks of Sharpness Aware MinimizationYoung In Kim, Andrea Agiollo, Pratiksha Agrawal, Johannes O. Royset et al.ICLR 2026 · 3 citations
- Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware MinimizationChaewon Moon, Dongkuk Si, Chulhee YunICLR 2026 · 1 citation
- Certification of Machine Learning Models via Directional SharpnessGefei Tan, Adrià Gascón, Sarah Meiklejohn, Mariana RaykovaUSENIX Security 2026
Builds on20
- Sharpness-aware Minimization for Efficiently Improving GeneralizationPierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam NeyshaburICLR 2021 · 1,861 citations
- Fantastic Generalization Measures and Where to Find ThemYiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan et al.ICLR 2020 · 705 citations
- The Pitfalls of Simplicity Bias in Neural NetworksHarshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain et al.NeurIPS 2020 · 503 citations
- Gradient Descent Maximizes the Margin of Homogeneous Neural NetworksKaifeng Lyu, Jian LiICLR 2020 · 402 citations
- ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural NetworksJungmin Kwon, Jeongseop Kim, Hyunseo Park, In Kwon ChoiICML 2021 · 385 citations
Related papers
- The alignment property of SGD noise and how it helps select flat minima: A stability analysisLei Wu, Mingze Wang, Weijie SuNeurIPS 2022 · 80 citations
- A Precise Characterization of SGD Stability Using Loss Surface GeometryGregory Dexter, Borja Ocejo, S. Sathiya Keerthi, Aman Gupta et al.ICLR 2024 · 2 citations
- Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better GeneralizationKaiyue Wen, Zhiyuan Li, Tengyu MaNeurIPS 2023 · 53 citations
- On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDTongcheng Zhang, Zhanpeng Zhou, Mingze Wang, Andi Han et al.AAAI 2026
- The Implicit Regularization of Dynamical Stability in Stochastic Gradient DescentLei Wu, Weijie J. SuICML 2023 · 41 citations
