Lune

EMNLP Findings2025

GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning

Abdessalam Ed-dib, Zhanibek Datbayev, Amine Mohamed Aboussalah

Abstract

Fine-tuning large language models (LLMs) is computationally expensive because it requires updating all model parameters. Low-Rank Adaptation (LoRA) reduces this cost by modifying a subset of weights, but selecting the appropriate rank introduces a trade-off: lower ranks improve efficiency at the expense of expressivity, while higher ranks enhance performance but increase computational burden. Existing adaptive LoRA methods lack a theoretical foundation to guide this trade-off optimally. We propose Geometric Low-Rank Adaptation (GeLoRA), a principled approach that estimates the intrinsic dimensionality of hidden data representations to adaptively select LoRA ranks. We show theoretically and empirically that the intrinsic dimension serves as a lower bound for the optimal rank of LoRA matrices, enabling a balance between efficiency and expressivity. Extensive experiments on GLUE, SQuAD (with DeBERTa), and MT-Bench (with LLaMA) demonstrate that GeLoRA consistently outperforms recent adaptive LoRA methods by up to +1.0%, while simultaneously reducing computational time by 13.5% to 64.2%, depending on the baseline, under the same parameter budget.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Deberta: decoding-Enhanced Bert with Disentangled Attention
  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • Deberta: decoding-Enhanced Bert with Disentangled Attention cites Language Models are Few-Shot Learners
  • QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
  • Less is More: Local Intrinsic Dimensions of Contextual Language Models cites Language Models are Few-Shot Learners
  • Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes cites LoRA: Low-Rank Adaptation of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Less is More: Local Intrinsic Dimensions of Contextual Language ModelsBenjamin Matthias Ruppik, Julius von Rohrscheidt, Carel van Niekerk, Michael Heck et al.NeurIPS 2025 · 1 citation
  2. Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap RegimesChuyan Zhang, Kefan Wang, Yun GuICCV 2025

Builds on18

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  4. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
  5. LoRA+: Efficient Low Rank Adaptation of Large ModelsSoufiane Hayou, Nikhil Ghosh, Bin YuICML 2024 · 388 citations

Related papers

  1. LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional AdaptationSeyedarmin Azizi, Souvik Kundu, Massoud PedramEMNLP Findings 2024 · 5 citations
  2. DenseLoRA: Dense Low-Rank Adaptation of Large Language ModelsLin Mu, Xiaoyu Wang, Li Ni, Yang Li et al.ACL 2025 · 3 citations
  3. RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large ModelsYilang Zhang, Bingcong Li, Georgios B. GiannakisNeurIPS 2025 · 9 citations
  4. MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-TuningPengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang et al.ACL 2024
  5. GeoLoRA: Geometric integration for parameter efficient fine-tuningSteffen Schotthöfer, Emanuele Zangrando, Gianluca Ceruti, Francesco Tudisco et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines