ACL Findings2025
CoLA: Collaborative Low-Rank Adaptation
Abstract
The scaling law of Large Language Models (LLMs) reveals a power-law relationship, showing diminishing return on performance as model scale increases. While training LLMs from scratch is resource-intensive, fine-tuning a pre-trained model for specific tasks has become a practical alternative. Full fine-tuning (FFT) achieves strong performance; however, it is computationally expensive and inefficient. Parameter-efficient fine-tuning (PEFT) methods, like LoRA, have been proposed to address these challenges by freezing the pre-trained model and adding lightweight task-specific modules. LoRA, in particular, has proven effective, but its application to multi-task scenarios is limited by interference between tasks. Recent approaches, such as Mixture-of-Experts (MOE) and asymmetric LoRA, have aimed to mitigate these issues but still struggle with sample scarcity and noise interference due to their fixed structure. In response, we propose CoLA, a more flexible LoRA architecture with an efficient initialization scheme, and introduces three collaborative strategies to enhance performance by better utilizing the quantitative relationships between matrices A and B. Our experiments demonstrate the effectiveness and robustness of CoLA, outperforming existing PEFT methods, especially in low-sample scenarios. Our data and code are fully publicly available 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating cites LoRA: Low-Rank Adaptation of Large Language Models
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Measuring Massive Multitask Language Understanding
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
- Beyond Student: An Asymmetric Network for Neural Network Inheritance cites LoRA: Low-Rank Adaptation of Large Language Models
- Beyond Student: An Asymmetric Network for Neural Network Inheritance cites Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating
- Beyond Student: An Asymmetric Network for Neural Network Inheritance cites Measuring Massive Multitask Language Understanding
- Beyond Student: An Asymmetric Network for Neural Network Inheritance cites Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities
- Beyond Student: An Asymmetric Network for Neural Network Inheritance cites AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization
- Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities cites LoRA: Low-Rank Adaptation of Large Language Models
- Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities cites AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization
- Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities cites MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites LoRA: Low-Rank Adaptation of Large Language Models
- AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization cites MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference
- MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference cites Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating
Cited by top-tier papers5
- AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache OptimizationZhonghua Jiang, Kui Chen, Kunxi Li, Keting Yin et al.AAAI 2026 · 4 citations
- Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model UpdatingYiyun Zhou, Zheqi Lv, Shengyu Zhang, Jingyuan ChenKDD 2025 · 1 citation
- Beyond Student: An Asymmetric Network for Neural Network InheritanceYiyun Zhou, Jingwei Shi, Mingjing Xu, Zhonghua Jiang et al.ICLR 2026 · 1 citation
- Collaborative Representation Learning for Alignment of Tactile, Language, and Vision ModalitiesYiyun Zhou, Mingjing Xu, Jingwei Shi, Quanjiang Li et al.AAAI 2026 · 1 citation
- MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context InferenceKunxi Li, Zhonghua Jiang, Zhouzhou Shen, Zhaode Wang et al.ACL 2025
Builds on24
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
- LEGAL-BERT: "Preparing the Muppets for Court'"Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras et al.EMNLP Findings 2020 · 1,073 citations
- The Flan Collection: Designing Data and Methods for Effective Instruction TuningShayne Longpre, Le Hou, Tu Vu, Albert Webson et al.ICML 2023 · 908 citations
Related papers
- MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task LearningDacao Zhang, Kun Zhang, Shimao Chu, Le Wu et al.ACL Findings 2025 · 10 citations
- MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task LearningYufei Ma, Zihan Liang, Huangyu Dai, Ben Chen et al.EMNLP 2024 · 4 citations
- HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA ExpertsMengqi Liao, Wei Chen, Junfeng Shen, Shengnan Guo et al.ICLR 2025
- LoRACoE: Improving Large Language Model via Composition-based LoRA ExpertGuanyu Li, Zhiheng Xi, Zhihao Zhang, Boyang Hong et al.EMNLP 2025
- SuLoRA: Subspace Low-Rank Adaptation for Parameter-Efficient Fine-TuningChenhao Ding, Jiangyang Li, Songlin Dong, Xinyuan Gao et al.ACL Findings 2025 · 2 citations
