ACL Findings2026
MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free
Yishu Lei, Shuwei He, Hu Jing, Dan Zhang, Xianlong Luo, Danxiang Zhu, Shikun Feng, Rui Liu, Jingzhou He, Yu Sun, Hua Wu, Haifeng Wang
Abstract
Extending the input modality of Large Language Models (LLMs) to the audio domain is essential for achieving comprehensive multimodal perception. However, it is well-known that acoustic information is intrinsically heterogeneous, entangling attributes such as speech, music, and environmental sounds. Existing methods typically rely on dense, parametershared adapters to model these diverse patterns, which induces gradient conflict during optimization, as parameter updates required for distinct attributes contradict each other. To address this limitation, we introduce the MoE-Adapter, a sparse Mixture-of-Experts (MoE) architecture designed to decouple acoustic information. Specifically, it employs a dynamic gating mechanism that routes audio tokens to specialized experts that capture complementary feature subspaces while retaining shared experts for global context, thereby mitigating gradient conflicts and enabling fine-grained feature learning. Comprehensive experiments show that the MoE-Adapter achieves superior performance on both audio semantic and paralinguistic tasks, consistently outperforming dense linear baselines with comparable computational costs. To facilitate future research, our code are publicly available at https:// github.com/Alittleegg/Eureka-Audio.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs cites Gradient Surgery for Multi-Task Learning
- Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging cites Gradient Surgery for Multi-Task Learning
- Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging cites Conflict-Averse Gradient Descent for Multi-task learning
- Scaling Vision with Sparse Mixture of Experts cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Conflict-Averse Gradient Descent for Multi-task learning cites Gradient Surgery for Multi-Task Learning
Cited by top-tier papers2
Builds on16
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
- Gradient Surgery for Multi-Task LearningTianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine et al.NeurIPS 2020 · 2,261 citations
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
- Scaling Vision with Sparse Mixture of ExpertsCarlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann et al.NeurIPS 2021 · 1,213 citations
- Conflict-Averse Gradient Descent for Multi-task learningBo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone et al.NeurIPS 2021 · 686 citations
Related papers
- Unveiling Multimodal Processing: Exploring Activation Patterns in Multimodal LLMs for Interpretability and EfficiencyChuan Wu, Meng Su, Youxuan Fang, Shaolin ZhuEMNLP Findings 2025
- SD-MoE: Spectral Decomposition for Effective Expert SpecializationRuijun Huang, Fang DONG(董方), Xin Zhang, Anrui Chen et al.ICML 2026 · 2 citations
- The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert LevelJeremy Herbst, Stefan Wermter, Jae Hee LeeICML 2026 · 9 citations
- UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity Mixture-of-ExpertsZhenyu Liu, Yunxin Li, Xuanyu Zhang, Qixun Teng et al.ACL 2026
- Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio EncodersWeiqiao Shan, Yuang Li, Yuhao Zhang, Yingfeng Luo et al.EMNLP 2025 · 1 citation
