Lune

ACL Findings2026

Hebbian-Guided Bi-Directional Rank Adaptation for Parameter-Efficient Fine-Tuning

Xing Zhao, Liu Yang, Xi-Le Zhao, Yueqi Yang, Tianqi Jiang

Abstract

Low-Rank Adaptation (LoRA) is a widely used method to fine-tune large language models, extremely reducing computational and storage costs. But its fixed-rank design cannot well capture the varying importance across different layers, limiting its flexibility. Dynamic rank allocation methods mitigate this issue by adaptively allocating ranks during training, but most of them focus solely on either rank pruning or expansion, leading to redundant parameterization or insufficient representational capacity. To address this problem, we introduce Hebbian-Guided Bi-Directional Rank Adaptation (HeBiRA), a novel framework that bidirectionally reallocates low-rank capacity using Hebbian-inspired importance estimation. HeBiRA computes the contribution of each rank direction by measuring the synergy between activations and output gradients, and adjusts the rank bi-directionally by pruning uninformative directions while expanding those in critical layers. This mechanism flexibly redistributes the rank budget during training and also can be applied to PEFT methods such as DoRA, HiRA, and QLoRA. Experiments on multiple benchmarks and theoretical analysis show that HeBiRA consistently improves performance over baselines.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
  • MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models cites QLoRA: Efficient Finetuning of Quantized LLMs
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on13

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  3. DoRA: Weight-Decomposed Low-Rank AdaptationShih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov et al.ICML 2024 · 820 citations
  4. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations
  5. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding SharingPengcheng He, Jianfeng Gao, Weizhu ChenICLR 2023 · 394 citations

Related papers

  1. FlexLoRA: Entropy-Guided Flexible Low-Rank AdaptationMuqing Liu, Chongjie Si, Yuheng JiaICLR 2026 · 3 citations
  2. DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank DistributionYulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao et al.ACL 2024 · 15 citations
  3. COBRA: Contribution-Based Bayesian Rank Allocation for Parameter-Efficient Fine-TuningHongcheng Ding, Xuanze Zhao, LIU XUANHUANG, Jing Jin et al.ICML 2026
  4. Low Kruskal-Rank AdaptationYixing Xu, Guanchen Li, Chao Li, Xuanwu Yin et al.ICML 2026
  5. Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language ModelsHao Zhang, Bo Huang, Zhenjia Li, Xi Xiao et al.EMNLP Findings 2025 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines