Lune

AAAI2026Top-tier venue

A Better Start: Sensitivity-Aware Warm-Up for Robust and Efficient Fine-Tuning

Yile Chen, Zeyi Wen, Jian Chen, Jin Huang

Abstract

As an essential component of fine-tuning, warm-up plays a crucial role in promoting stability and generalization. Many studies have examined its underlying mechanisms from different aspects. However, most of the studies focus on incorporating these insights into optimizers to reduce the reliance on warm-up. Little attention has been paid to addressing the inherent limitations of the warm-up itself, which restricts its effectiveness. In this work, we revisit warm-up from a loss landscape perspective and identify several limitations with existing warm-up, including: (1) susceptibility to nearby suboptimal traps, (2) sensitivity to hyperparameters and random seeds, and (3) inefficiency during the early stages of training. To overcome these limitations, we propose Sensitivity-Aware Warm-Up (SAWU), a lightweight and adaptive strategy that dynamically leverages learning sensitivity during warm-up to guide updates toward better and more stable basins. In addition, SAWU also introduces an adaptive scheduling mechanism and phase transition strategy across warm-up, stable, and decay phases to further enhance robustness and efficiency. Extensive experiments on various downstream tasks show that SAWU significantly outperforms the vanilla method (e.g., average 3.43% improvement on RoBerta). Moreover, SAWU can be easily combined with various optimizers and remains effective even when warm-up-based methods fail (e.g, it lifts RAdam from 49.46% to 91.78% on qnli. Thanks to its lightweight nature, SAWU introduces minimal overhead and even reduces training time by over 5% compared to other methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Why Warmup the Learning Rate? Underlying Mechanisms and Improvements cites On the Variance of the Adaptive Learning Rate and Beyond
  • Small-scale proxies for large-scale Transformer training instabilities cites On the Variance of the Adaptive Learning Rate and Beyond
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on7

  1. On the Variance of the Adaptive Learning Rate and BeyondLiyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen et al.ICLR 2020 · 2,210 citations
  2. Large Batch Optimization for Deep Learning: Training BERT in 76 minutesYang You, Jing Li, Sashank J. Reddi, Jonathan Hseu et al.ICLR 2020 · 1,170 citations
  3. Small-scale proxies for large-scale Transformer training instabilitiesMitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie E. Everett et al.ICLR 2024 · 162 citations
  4. On the Generalization Benefit of Noise in Stochastic Gradient DescentSamuel L. Smith, Erich Elsen, Soham DeICML 2020 · 122 citations
  5. Why Warmup the Learning Rate? Underlying Mechanisms and ImprovementsDayal Singh Kalra, Maissam BarkeshliNeurIPS 2024 · 87 citations

Related papers

  1. Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-TuningKazuki Yano, Shun Kiyono, Sosuke Kobayashi, Sho Takase et al.ICLR 2026 · 13 citations
  2. ROSE: Robust Selective Fine-tuning for Pre-trained Language ModelsLan Jiang, Hao Zhou, Yankai Lin, Peng Li et al.EMNLP 2022 · 5 citations
  3. Sensitivity-Aware Visual Parameter-Efficient Fine-TuningHaoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao et al.ICCV 2023 · 97 citations
  4. Stochastic Fine-Tuning of Language Models Using Masked GradientsMohammad Akbar-Tajari, Mohammad Taher PilehvarEMNLP Findings 2024 · 2 citations
  5. Beyond Static Allocation: Dynamic Sensitivity-Aware Fine-Tuning for Vision TransformersYuanyang Cao, Xichun Liu, Fuwei Zhang, Shangqi Deng et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines