ACL Findings2026
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
Boyan Shi, Wei Chen, Shuyuan Zhao, Junfeng Shen, Shengnan Guo, Shaojiang Wang, Huaiyu Wan
Abstract
The combination of Mixture-of-Experts (MoE) and Low-Rank Adaptation (LoRA) has shown significant potential for enhancing the multitask learning capabilities of Large Language Models. However, existing methods face two primary challenges: (1)Imprecise Routing in the current MoE-LoRA method fails to explicitly match input semantics with expert capabilities, leading to weak expert specialization.
(2)Uniform weight fusion strategies struggle to provide adaptive update strengths, overlooking the varying complexity of different tasks. To address these limitations, we propose SAMoRA (Semantic-Aware Mixture of LoRA Experts), a novel parameter-efficient fine-tuning framework tailored for task-adaptive learning. Specifically, A Semantic-Aware Router is proposed to explicitly align textual semantics with the most suitable experts for precise routing. A Task-Adaptive Scaling mechanism is designed to regulate expert contributions based on specific task requirements dynamically. In addition, a novel regularization objective is proposed to jointly promote expert specialization and effective scaling. Extensive experiments on multiple multi-task benchmarks demonstrate that SAMoRA significantly outperforms the state-of-the-art methods and holds excellent task generalization capabilities. Code is available at https://github.com/boyan-code/ SAMoRA
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Finetuned Language Models are Zero-Shot Learners cites PIQA: Reasoning about Physical Commonsense in Natural Language
- Finetuned Language Models are Zero-Shot Learners cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Finetuned Language Models are Zero-Shot Learners cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Builds on12
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
Related papers
- Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-task LearningZiyu Zhao, Yixiao Zhou, Xin Yu, Zhi Zhang et al.KDD 2026 · 13 citations
- When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical ApplicationsQidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu et al.SIGIR 2024 · 89 citations
- D2MoRA: Diversity-Regulated Asymmetric MoE-LoRA Decomposition for Efficient Multi-Task AdaptationJianhui Zuo, Xuemeng Song, Haokun Wen, Meng Liu et al.AAAI 2026
- TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language ModelsLin Mu, Haiyang Wang, Li Ni, Lei Sang et al.ACL 2026 · 1 citation
- HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA ExpertsMengqi Liao, Wei Chen, Junfeng Shen, Shengnan Guo et al.ICLR 2025
