GoRA: Gradient-driven Adaptive Low Rank Adaptation
Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, Lei Chen
Abstract
Low-Rank Adaptation (LoRA) is a crucial method for efficiently fine-tuning large language models (LLMs), with its effectiveness influenced by two key factors: rank selection and weight initialization. While numerous LoRA variants have been proposed to improve performance by addressing one of these aspects, they often compromise usability or computational efficiency. In this paper, we analyze and identify the core limitations of existing approaches and propose a novel framework--GoRA (Gradient-driven Adaptive Low Rank Adaptation)--that simultaneously adapts both the rank and initialization strategy within a unified framework. GoRA leverages gradient information during training to dynamically assign optimal ranks and initialize low-rank adapter weights in an adaptive manner. To our knowledge, GoRA is the first method that not only addresses the limitations of prior approaches--which often focus on either rank selection or initialization in isolation--but also unifies both aspects within a single framework, enabling more effective and efficient adaptation. Extensive experiments across various architectures and modalities show that GoRA consistently outperforms existing LoRA-based methods while preserving the efficiency of vanilla LoRA. For example, when fine-tuning Llama3.1-8B-Base for mathematical reasoning, GoRA achieves a 5.13-point improvement over standard LoRA and even outperforms full fine-tuning by 2.05 points under high-rank settings. Code is available at: https://github.com/hhnqqq/MyTransformers.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MTNL: A Unified Modeling Perspective for Enhancing Tensor Network Learning cites LoRA: Low-Rank Adaptation of Large Language Models
- COBRA: Contribution-Based Bayesian Rank Allocation for Parameter-Efficient Fine-Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation cites LoRA: Low-Rank Adaptation of Large Language Models
- Can Muon Fine-tune Adam-Pretrained Models? cites LoRA: Low-Rank Adaptation of Large Language Models
- Can Muon Fine-tune Adam-Pretrained Models? cites WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions
- Can Muon Fine-tune Adam-Pretrained Models? cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers5
- MTNL: A Unified Modeling Perspective for Enhancing Tensor Network LearningJunhua Zeng, Yuning Qiu, Binghua Li, Chao Li et al.ICML 2026
- COBRA: Contribution-Based Bayesian Rank Allocation for Parameter-Efficient Fine-TuningHongcheng Ding, Xuanze Zhao, LIU XUANHUANG, Jing Jin et al.ICML 2026
- CSPLoRA: Confidence-Guided Structure Planning for Low-Rank AdaptationHuiming Ding, Xiaochen Li, Jianhui Ma, Xu An et al.ICML 2026
- GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream AdaptationSungmin Kang, Jisoo Kim, Salman Avestimehr, Sunwoo LeeAAAI 2026
- Can Muon Fine-tune Adam-Pretrained Models?Xingyu Qu, Peigeng Huang, Samuel HorváthICML 2026
Builds on21
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsCan Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng et al.ICLR 2024 · 1,206 citations
Related papers
- LeLoRA: Learnable Low-Rank Adaptation of Large Language ModelsXiaoling Zhou, Mingjie Zhang, Zhemg Lee, Wei Ye et al.ACL 2026
- PLoRA: Efficient Concurrent LoRA Training for Large Language ModelsMinghao Yan, Zhuang Wang, Zhen Jia, Shivaram Venkataraman et al.ICML 2026 · 5 citations
- DenseLoRA: Dense Low-Rank Adaptation of Large Language ModelsLin Mu, Xiaoyu Wang, Li Ni, Yang Li et al.ACL 2025 · 3 citations
- AIRA: Activation-Informed Low-Rank Adaptation for Large ModelsLujun Li, Dezhi Li, Cheng Lin, Wei Li et al.ICCV 2025
- TLoRA: Task-aware Low Rank Adaptation of Large Language ModelsWeicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie ZhangACL 2026
