ACL Findings2025
LoRMA: Low-Rank Multiplicative Adaptation for LLMs
Harsh Bihany, Shubham Patel, Ashutosh Modi
Abstract
Large Language Models have shown remarkable capabilities in the NLP domain. Their effectiveness can mainly be attributed to their ability to adapt to an array of downstream tasks. However, generally, full fine-tuning is a computationally expensive job. To mitigate this, many techniques have been developed that prime efficiency, a prominent one being Low-Rank Adaptation (LoRA). However, LoRA and its variants employ re-parametrized additive updates. In this paper, we propose Low-Rank Multiplicative Adaptation (LoRMA), which shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. We tackle challenges such as computational complexity and rank bottleneck of matrix multiplication by effectively re-ordering operations and introducing rank inflation strategies. We conduct extensive experiments to demonstrate the effectiveness of our approach in terms of various evaluation metrics.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on16
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
- DoRA: Weight-Decomposed Low-Rank AdaptationShih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov et al.ICML 2024 · 820 citations
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations
- A Survey on In-context LearningQingxiu Dong, Lei Li, Damai Dai, Ce Zheng et al.EMNLP 2024 · 479 citations
Related papers
- HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language ModelsQiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang et al.ICLR 2025
- Polynomial Expansion Rank Adaptation: Enhancing Low-Rank Fine-Tuning with High-Order InteractionsWenhao Zhang, Lin Mu, Li Ni, Peiquan Jin et al.ACL Findings 2026 · 1 citation
- Low Kruskal-Rank AdaptationYixing Xu, Guanchen Li, Chao Li, Xuanwu Yin et al.ICML 2026
- SuLoRA: Subspace Low-Rank Adaptation for Parameter-Efficient Fine-TuningChenhao Ding, Jiangyang Li, Songlin Dong, Xinyuan Gao et al.ACL Findings 2025 · 2 citations
- PLoRA: Efficient Concurrent LoRA Training for Large Language ModelsMinghao Yan, Zhuang Wang, Zhen Jia, Shivaram Venkataraman et al.ICML 2026 · 5 citations
