CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

Large language models (LLMs) achieve remarkable performance on diverse downstream and domain-specific tasks via parameterefficient fine-tuning (PEFT). However, existing PEFT methods, particularly MoE-LoRA architectures, suffer from limited parameter efficiency and coarse-grained adaptation due to the proliferation of LoRA experts and instancelevel routing. To address these issues, we propose Core Space Mixture of LoRA (CoMoL), a novel MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation. Specifically, CoMoL introduces two key components: Core Space Experts and Core Space Routing. Core Space Experts store each expert in a compact core matrix, preserving diversity while controlling parameter growth. Core Space Routing dynamically selects and activates the appropriate core experts for each token, enabling finegrained, input-adaptive routing. Activated core experts are then merged via a soft-merging strategy into a single core expert, which is combined with a shared LoRA to form a specialized LoRA module. Besides, the routing network is projected into the same low-rank space as the LoRA matrices, further reducing parameter overhead without compromising expressiveness. Extensive experiments demonstrate that CoMoL preserves the adaptability of MoE-LoRA architectures while achieving parameter efficiency comparable to Standard LoRA, consistently outperforming existing methods across multiple tasks. Our code is available at https://github.com/DCDmllm/CoMoL.