Lune

ACL Findings2026

G-LoRA: Global-Local Decoupled Low-Rank Adaptation

Jiahao Xiong, Yihong Huang, Yihe Liu, Xianming Hu, Hongbo Zhao, Kai Zhang

Abstract

Low-Rank Adaptation (LoRA) has achieved remarkable progress in improving the finetuning efficiency and downstream performance of large language models (LLMs). Although prior work has recognized that different weight update matrices ∆W exhibit varying importance and therefore should be allocated different ranks, parameters within the same update matrix are still typically constrained to a uniform rank configuration, neglecting finegrained parameter-level heterogeneity. To address this limitation, we propose G-LoRA (Global-Local Decoupled LoRA), which decomposes each update matrix into global and local adapters. The key idea is to reorganize the rows and columns of the update matrix using a first-order Taylor approximation of parameter importance, such that highly influential parameters are clustered into a local sub-block of ∆W. During training, the local adapter then focuses on this high-importance sub-region and is allocated a higher rank, whereas the global adapter captures the residual updates for the entire update matrix with relatively lower rank. By allocating higher representational capacity to more critical parameters, G-LoRA enables more efficient utilization of model resources. Extensive evaluations on benchmarks spanning commonsense reasoning, mathematical reasoning, and code generation demonstrate that G-LoRA achieves up to 2.7% absolute accuracy improvement over LoRA and its variants, validating its effectiveness for LLM fine-tuning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • DoRA: Weight-Decomposed Low-Rank Adaptation cites LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on20

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  3. DoRA: Weight-Decomposed Low-Rank AdaptationShih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov et al.ICML 2024 · 820 citations
  4. PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language ModelsFanxu Meng, Zhaohui Wang, Muhan ZhangNeurIPS 2024 · 374 citations
  5. LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language ModelsZhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu et al.EMNLP 2023 · 200 citations

Related papers

  1. Localized Low-Rank Adaptation within Clustered Parameter SubspacesJiahao Xiong, Yihe Liu, Xianming Hu, Hongbo Zhao et al.ACL 2026
  2. Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language ModelsHao Zhang, Bo Huang, Zhenjia Li, Xi Xiao et al.EMNLP Findings 2025 · 4 citations
  3. CSPLoRA: Confidence-Guided Structure Planning for Low-Rank AdaptationHuiming Ding, Xiaochen Li, Jianhui Ma, Xu An et al.ICML 2026
  4. TLoRA: Task-aware Low Rank Adaptation of Large Language ModelsWeicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie ZhangACL 2026
  5. ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-TuningYilang Zhang, Xiaodong Yang, Yiwei Cai, Georgios B. GiannakisICML 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines