ACL Findings2026
G-LoRA: Global-Local Decoupled Low-Rank Adaptation
Jiahao Xiong, Yihong Huang, Yihe Liu, Xianming Hu, Hongbo Zhao, Kai Zhang
Abstract
Low-Rank Adaptation (LoRA) has achieved remarkable progress in improving the finetuning efficiency and downstream performance of large language models (LLMs). Although prior work has recognized that different weight update matrices ∆W exhibit varying importance and therefore should be allocated different ranks, parameters within the same update matrix are still typically constrained to a uniform rank configuration, neglecting finegrained parameter-level heterogeneity. To address this limitation, we propose G-LoRA (Global-Local Decoupled LoRA), which decomposes each update matrix into global and local adapters. The key idea is to reorganize the rows and columns of the update matrix using a first-order Taylor approximation of parameter importance, such that highly influential parameters are clustered into a local sub-block of ∆W. During training, the local adapter then focuses on this high-importance sub-region and is allocated a higher rank, whereas the global adapter captures the residual updates for the entire update matrix with relatively lower rank. By allocating higher representational capacity to more critical parameters, G-LoRA enables more efficient utilization of model resources. Extensive evaluations on benchmarks spanning commonsense reasoning, mathematical reasoning, and code generation demonstrate that G-LoRA achieves up to 2.7% absolute accuracy improvement over LoRA and its variants, validating its effectiveness for LLM fine-tuning.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
- DoRA: Weight-Decomposed Low-Rank Adaptation cites LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
Builds on20
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- DoRA: Weight-Decomposed Low-Rank AdaptationShih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov et al.ICML 2024 · 820 citations
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language ModelsFanxu Meng, Zhaohui Wang, Muhan ZhangNeurIPS 2024 · 374 citations
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language ModelsZhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu et al.EMNLP 2023 · 200 citations
Related papers
- Localized Low-Rank Adaptation within Clustered Parameter SubspacesJiahao Xiong, Yihe Liu, Xianming Hu, Hongbo Zhao et al.ACL 2026
- Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language ModelsHao Zhang, Bo Huang, Zhenjia Li, Xi Xiao et al.EMNLP Findings 2025 · 4 citations
- CSPLoRA: Confidence-Guided Structure Planning for Low-Rank AdaptationHuiming Ding, Xiaochen Li, Jianhui Ma, Xu An et al.ICML 2026
- TLoRA: Task-aware Low Rank Adaptation of Large Language ModelsWeicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie ZhangACL 2026
- ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-TuningYilang Zhang, Xiaodong Yang, Yiwei Cai, Georgios B. GiannakisICML 2026 · 1 citation
