Lune

ICLR2025Top-tier venue

Sharpness-Aware Minimization: General Analysis and Improved Rates

Dimitris Oikonomou, Nicolas Loizou

Abstract

Sharpness-Aware Minimization (SAM) has emerged as a powerful method for improving generalization in machine learning models by minimizing the sharpness of the loss landscape. However, despite its success, several important questions regarding the convergence properties of SAM in non-convex settings are still open, including the benefits of using normalization in the update rule, the dependence of the analysis on the restrictive bounded variance assumption, and the convergence guarantees under different sampling strategies. To address these questions, in this paper, we provide a unified analysis of SAM and its unnormalized variant (USAM) under one single flexible update rule (Unified SAM), and we present convergence results of the new algorithm under a relaxed and more natural assumption on the stochastic noise. Our analysis provides convergence guarantees for SAM under different step size selections for non-convex problems and functions that satisfy the Polyak-Lojasiewicz (PL) condition (a non-convex generalization of strongly convex functions). The proposed theory holds under the arbitrary sampling paradigm, which includes importance sampling as special case, allowing us to analyze variants of SAM that were never explicitly considered in the literature. Experiments validate the theoretical findings and further demonstrate the practical effectiveness of Unified SAM in training deep neural networks for image classification tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler cites Adversarial Weight Perturbation Helps Robust Generalization
  • Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler cites Fantastic Generalization Measures and Where to Find Them
  • Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler cites Towards Understanding Sharpness-Aware Minimization
  • Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler cites Sharpness-aware Minimization for Efficiently Improving Generalization
  • Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise cites Towards Understanding Sharpness-Aware Minimization
  • Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise cites Sharpness-aware Minimization for Efficiently Improving Generalization
  • Bilevel Optimization for Adversarial Learning Problems: Sharpness, Generation, and Beyond cites Adversarial Weight Perturbation Helps Robust Generalization
  • Bilevel Optimization for Adversarial Learning Problems: Sharpness, Generation, and Beyond cites Fantastic Generalization Measures and Where to Find Them
  • Bilevel Optimization for Adversarial Learning Problems: Sharpness, Generation, and Beyond cites Towards Understanding Sharpness-Aware Minimization
  • Bilevel Optimization for Adversarial Learning Problems: Sharpness, Generation, and Beyond cites Sharpness-aware Minimization for Efficiently Improving Generalization
  • Understanding SAM through Minimax Perspective cites Towards Understanding Sharpness-Aware Minimization
  • Understanding SAM through Minimax Perspective cites Sharpness-aware Minimization for Efficiently Improving Generalization
  • Towards Understanding Sharpness-Aware Minimization cites Adversarial Weight Perturbation Helps Robust Generalization
  • Towards Understanding Sharpness-Aware Minimization cites Fantastic Generalization Measures and Where to Find Them
  • Towards Understanding Sharpness-Aware Minimization cites Sharpness-aware Minimization for Efficiently Improving Generalization
  • Sharpness-aware Minimization for Efficiently Improving Generalization cites Fantastic Generalization Measures and Where to Find Them
  • Improving Model-Based Reinforcement Learning by Converging to Flatter Minima cites Adversarial Weight Perturbation Helps Robust Generalization
  • Improving Model-Based Reinforcement Learning by Converging to Flatter Minima cites Fantastic Generalization Measures and Where to Find Them
  • Improving Model-Based Reinforcement Learning by Converging to Flatter Minima cites Towards Understanding Sharpness-Aware Minimization
  • Improving Model-Based Reinforcement Learning by Converging to Flatter Minima cites Sharpness-aware Minimization for Efficiently Improving Generalization
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Improving Model-Based Reinforcement Learning by Converging to Flatter MinimaShrinivas Ramasubramanian, Benjamin Freed, Alexandre Capone, Jeff G. SchneiderNeurIPS 2025 · 3 citations
  2. Unveiling m-Sharpness Through the Structure of Stochastic Gradient NoiseHaocheng Luo, Mehrtash Harandi, Dinh Phung, Trung LeNeurIPS 2025 · 2 citations
  3. Bilevel Optimization for Adversarial Learning Problems: Sharpness, Generation, and BeyondRisheng Liu, Zhu Liu, Weihao Mao, Wei Yao et al.NeurIPS 2025 · 2 citations
  4. Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded SchedulerDimitris Oikonomou, Nicolas LoizouICML 2026
  5. Understanding SAM through Minimax PerspectiveYing Chen, Aoxi Li, Javad LavaeiICML 2026

Builds on17

  1. Sharpness-aware Minimization for Efficiently Improving GeneralizationPierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam NeyshaburICLR 2021 · 1,861 citations
  2. Adversarial Weight Perturbation Helps Robust GeneralizationDongxian Wu, Shu-Tao Xia, Yisen WangNeurIPS 2020 · 917 citations
  3. Fantastic Generalization Measures and Where to Find ThemYiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan et al.ICLR 2020 · 705 citations
  4. Bag of Tricks for Adversarial TrainingTianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su et al.ICLR 2021 · 298 citations
  5. Towards Understanding Sharpness-Aware MinimizationMaksym Andriushchenko, Nicolas FlammarionICML 2022 · 190 citations

Related papers

  1. Sharpness-Aware Minimization Can Hallucinate MinimizersChanwoong Park, Uijeong Jang, Ernest Ryu, Insoon YangICML 2026
  2. On Saddle Point Avoidance and Stationary Distribution of Sharpness-Aware MinimizationTao Sun, Fan Jia, Bao WangKDD 2026 · 1 citation
  3. Friendly Sharpness-Aware MinimizationTao Li, Pan Zhou, Zhengbao He, Xinwen Cheng et al.CVPR 2024
  4. Beyond Sharpness: The Role of Nonuniformity in GeneralizationYingcong Zhou, Pingfan Wu, Li Wang, Zhiguo Fu et al.AAAI 2026
  5. The Crucial Role of Normalization in Sharpness-Aware MinimizationYan Dai, Kwangjun Ahn, Suvrit SraNeurIPS 2023 · 36 citations
Dusk over the sea between two cliffs drawn in fine vertical lines