Lune

ACL Findings2025

CoLA: Collaborative Low-Rank Adaptation

Yiyun Zhou, Chang Yao, Jingyuan Chen

Abstract

The scaling law of Large Language Models (LLMs) reveals a power-law relationship, showing diminishing return on performance as model scale increases. While training LLMs from scratch is resource-intensive, fine-tuning a pre-trained model for specific tasks has become a practical alternative. Full fine-tuning (FFT) achieves strong performance; however, it is computationally expensive and inefficient. Parameter-efficient fine-tuning (PEFT) methods, like LoRA, have been proposed to address these challenges by freezing the pre-trained model and adding lightweight task-specific modules. LoRA, in particular, has proven effective, but its application to multi-task scenarios is limited by interference between tasks. Recent approaches, such as Mixture-of-Experts (MOE) and asymmetric LoRA, have aimed to mitigate these issues but still struggle with sample scarcity and noise interference due to their fixed structure. In response, we propose CoLA, a more flexible LoRA architecture with an efficient initialization scheme, and introduces three collaborative strategies to enhance performance by better utilizing the quantitative relationships between matrices A and B. Our experiments demonstrate the effectiveness and robustness of CoLA, outperforming existing PEFT methods, especially in low-sample scenarios. Our data and code are fully publicly available 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating cites LoRA: Low-Rank Adaptation of Large Language Models
  • The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
  • The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Measuring Massive Multitask Language Understanding
  • The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
  • Beyond Student: An Asymmetric Network for Neural Network Inheritance cites LoRA: Low-Rank Adaptation of Large Language Models
  • Beyond Student: An Asymmetric Network for Neural Network Inheritance cites Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating
  • Beyond Student: An Asymmetric Network for Neural Network Inheritance cites Measuring Massive Multitask Language Understanding
  • Beyond Student: An Asymmetric Network for Neural Network Inheritance cites Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities
  • Beyond Student: An Asymmetric Network for Neural Network Inheritance cites AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization
  • Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities cites LoRA: Low-Rank Adaptation of Large Language Models
  • Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities cites AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization
  • Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities cites MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites LoRA: Low-Rank Adaptation of Large Language Models
  • AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization cites MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference
  • MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference cites Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache OptimizationZhonghua Jiang, Kui Chen, Kunxi Li, Keting Yin et al.AAAI 2026 · 4 citations
  2. Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model UpdatingYiyun Zhou, Zheqi Lv, Shengyu Zhang, Jingyuan ChenKDD 2025 · 1 citation
  3. Beyond Student: An Asymmetric Network for Neural Network InheritanceYiyun Zhou, Jingwei Shi, Mingjing Xu, Zhonghua Jiang et al.ICLR 2026 · 1 citation
  4. Collaborative Representation Learning for Alignment of Tactile, Language, and Vision ModalitiesYiyun Zhou, Mingjing Xu, Jingwei Shi, Quanjiang Li et al.AAAI 2026 · 1 citation
  5. MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context InferenceKunxi Li, Zhonghua Jiang, Zhouzhou Shen, Zhaode Wang et al.ACL 2025

Builds on24

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
  4. LEGAL-BERT: "Preparing the Muppets for Court'"Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras et al.EMNLP Findings 2020 · 1,073 citations
  5. The Flan Collection: Designing Data and Methods for Effective Instruction TuningShayne Longpre, Le Hou, Tu Vu, Albert Webson et al.ICML 2023 · 908 citations

Related papers

  1. MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task LearningDacao Zhang, Kun Zhang, Shimao Chu, Le Wu et al.ACL Findings 2025 · 10 citations
  2. MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task LearningYufei Ma, Zihan Liang, Huangyu Dai, Ben Chen et al.EMNLP 2024 · 4 citations
  3. HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA ExpertsMengqi Liao, Wei Chen, Junfeng Shen, Shengnan Guo et al.ICLR 2025
  4. LoRACoE: Improving Large Language Model via Composition-based LoRA ExpertGuanyu Li, Zhiheng Xi, Zhihao Zhang, Boyang Hong et al.EMNLP 2025
  5. SuLoRA: Subspace Low-Rank Adaptation for Parameter-Efficient Fine-TuningChenhao Ding, Jiangyang Li, Songlin Dong, Xinyuan Gao et al.ACL Findings 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines