Lune

NeurIPS2025Top-tier venue

Asymptotic theory of SGD with a general learning-rate

Or Goldreich, Ziyang Wei, Soham Bonnerjee, Jiaqi Li, Wei Biao Wu

Abstract

Stochastic gradient descent (SGD) with polynomially decaying step-sizes has long underpinned theoretical analyses, yielding a broad spectrum of statistically attractive guarantees. Yet in practice, such schedules find rare use due to their prohibitively slow convergence, revealing a persistent gap between theory and empirical performance. In this paper, we introduce a unified framework that quantifies the uncertainty of online SGD under arbitrary learning-rate choices. In particular, we provide the first comprehensive convergence characterizations for two widely used but theoretically under-examined schemes-cyclical learning rates and linear decay to zero. Our results not only explain the observed behavior of these schedules but also facilitate principled tools for statistical inference and algorithm design. All theoretical findings are corroborated by extensive simulations across diverse settings.

Stochastic Gradient Descent (SGD) has gained popularity in modern machine learning since the seminal work of Robbins and Monro [1951]. While its theoretical foundations are well established, the literature has largely focused on two standard step-size choices: constant step-sizes, which provide exponentially fast convergence to a biased stationary distribution and allow straightforward tuning, and polynomially decaying step-sizes, typically of the form η t ≍ t -α , which offer statistical guarantees such as consistency and asymptotic normality [

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs cites An empirical analysis of compute-optimal large language model training
  • Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs cites Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
  • Sharp asymptotic theory for Q-learning with LD2Z learning rate and its generalization cites An empirical analysis of compute-optimal large language model training
  • Sharp asymptotic theory for Q-learning with LD2Z learning rate and its generalization cites Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
  • Sharp asymptotic theory for Q-learning with LD2Z learning rate and its generalization cites Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Sharp asymptotic theory for Q-learning with LD2Z learning rate and its generalizationSoham Bonnerjee, Zhipeng Lou, Wei Biao WuICLR 2026

Builds on3

  1. An empirical analysis of compute-optimal large language model trainingJordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya et al.NeurIPS 2022 · 566 citations
  2. Scaling Laws and Compute-Optimal Training Beyond Fixed Training DurationsAlexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal et al.NeurIPS 2024 · 168 citations
  3. Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMsShane Bergsma, Nolan Simran Dey, Gurpreet Gosal, Gavia Gray et al.ICLR 2025 · 1 citation

Related papers

  1. On the Convergence of Step Decay Step-Size for Stochastic OptimizationXiaoyu Wang, Sindri Magnússon, Mikael JohanssonNeurIPS 2021 · 33 citations
  2. Statistical Guarantees for High-Dimensional Stochastic Gradient DescentJiaqi Li, Zhipeng Lou, Johannes Schmidt-Hieber, Wei Biao WuNeurIPS 2025 · 3 citations
  3. A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and PerformanceXiaoyu Li, Zhenxun Zhuang, Francesco OrabonaICML 2021 · 29 citations
  4. Learning Rate Annealing Improves Tuning Robustness in Stochastic OptimizationAmit Attia, Tomer KorenICML 2026
  5. Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear RegressionJingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu et al.ICML 2022 · 38 citations
Dusk over the sea between two cliffs drawn in fine vertical lines