Lune

ICML2025Top-tier venue

Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects

Ke Liang Xiao, Noah Marshall, Atish Agarwala, Elliot Paquette

Abstract

In recent years, SIGNSGD has garnered interest as both a practical optimizer as well as a simple model to understand adaptive optimizers like ADAM. Though there is a general consensus that SIGNSGD acts to precondition optimization and reshapes noise, quantitatively understanding these effects in theoretically solvable settings remains difficult. We present an analysis of SIGNSGD in a high-dimensional limit, and derive a limiting SDE and ODE to describe the risk. Using this framework we quantify four effects of SIGNSGD: effective learning rate, noise compression, diagonal preconditioning, and gradient noise reshaping. Our analysis is consistent with experimental observations but moves beyond that by quantifying the dependence of these effects on the data and noise distributions. We conclude with a conjecture on how these results might be extended to ADAM.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Per-example Gradients: a New Frontier for Understanding and Improving Optimizers cites Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
  • Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise cites Why are Adaptive Methods Good for Attention Models?
  • Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise cites On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
  • Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law cites Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
  • Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law cites Symbolic Discovery of Optimization Algorithms
  • Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law cites On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
  • On the Interaction of Batch Noise, Adaptivity, and Compression, under (L_0,L_1)-Smoothness: An SDE Approach cites Why are Adaptive Methods Good for Attention Models?
  • Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective cites Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
  • Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective cites On the Interaction of Batch Noise, Adaptivity, and Compression, under (L_0,L_1)-Smoothness: An SDE Approach
  • Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective cites On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
  • On the SDEs and Scaling Rules for Adaptive Gradient Algorithms cites Why are Adaptive Methods Good for Attention Models?
  • Why Transformers Need Adam: A Hessian Perspective cites Why are Adaptive Methods Good for Attention Models?
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's LawFrederik Kunstner, Francis BachNeurIPS 2025 · 21 citations
  2. On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)(L_0,L_1)-Smoothness: An SDE ApproachEnea Monzio Compagnoni, Rustem Islamov, Frank Proske, Aurelien Lucchi et al.ICML 2026 · 4 citations
  3. Per-example Gradients: a New Frontier for Understanding and Improving OptimizersVincent Roulet, Atish AgarwalaICML 2026 · 2 citations
  4. Adaptive Methods Are Preferable in High Privacy Settings: An SDE PerspectiveEnea Monzio Compagnoni, Alessandro Stanghellini, Rustem Islamov, Aurélien Lucchi et al.ICLR 2026 · 2 citations
  5. Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of NoiseEnea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske et al.ICLR 2025

Builds on9

  1. Symbolic Discovery of Optimization AlgorithmsXiangning Chen, Chen Liang, Da Huang, Esteban Real et al.NeurIPS 2023 · 734 citations
  2. Why are Adaptive Methods Good for Attention Models?Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim et al.NeurIPS 2020 · 397 citations
  3. Why Transformers Need Adam: A Hessian PerspectiveYushun Zhang, Congliang Chen, Tian Ding, Ziniu Li et al.NeurIPS 2024 · 149 citations
  4. On the SDEs and Scaling Rules for Adaptive Gradient AlgorithmsSadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, Sanjeev AroraNeurIPS 2022 · 125 citations
  5. Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High DimensionsCourtney Paquette, Elliot Paquette, Ben Adlam, Jeffrey PenningtonNeurIPS 2022 · 22 citations

Related papers

  1. On the Optimization and Generalization of Two-layer Transformers with Sign Gradient DescentBingrui Li, Wei Huang, Andi Han, Zhanpeng Zhou et al.ICLR 2025
  2. Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?Jihwan Kim, Dogyoon Song, Chulhee YunICLR 2026
  3. Stochastic Sign Descent Methods: New Algorithms and Better TheoryMher Safaryan, Peter RichtárikICML 2021 · 70 citations
  4. SGD with Adaptive Preconditioning: Unified Analysis and Momentum AccelerationDmitry KovalevICLR 2026 · 13 citations
  5. Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer ManifoldXinghan Li, Haodong Wen, Kaifeng LyuNeurIPS 2025 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines