Lune

ACL Findings2026

MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free

Yishu Lei, Shuwei He, Hu Jing, Dan Zhang, Xianlong Luo, Danxiang Zhu, Shikun Feng, Rui Liu, Jingzhou He, Yu Sun, Hua Wu, Haifeng Wang

Abstract

Extending the input modality of Large Language Models (LLMs) to the audio domain is essential for achieving comprehensive multimodal perception. However, it is well-known that acoustic information is intrinsically heterogeneous, entangling attributes such as speech, music, and environmental sounds. Existing methods typically rely on dense, parametershared adapters to model these diverse patterns, which induces gradient conflict during optimization, as parameter updates required for distinct attributes contradict each other. To address this limitation, we introduce the MoE-Adapter, a sparse Mixture-of-Experts (MoE) architecture designed to decouple acoustic information. Specifically, it employs a dynamic gating mechanism that routes audio tokens to specialized experts that capture complementary feature subspaces while retaining shared experts for global context, thereby mitigating gradient conflicts and enabling fine-grained feature learning. Comprehensive experiments show that the MoE-Adapter achieves superior performance on both audio semantic and paralinguistic tasks, consistently outperforming dense linear baselines with comparable computational costs. To facilitate future research, our code are publicly available at https:// github.com/Alittleegg/Eureka-Audio.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs cites Gradient Surgery for Multi-Task Learning
  • Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging cites Gradient Surgery for Multi-Task Learning
  • Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging cites Conflict-Averse Gradient Descent for Multi-task learning
  • Scaling Vision with Sparse Mixture of Experts cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  • Conflict-Averse Gradient Descent for Multi-task learning cites Gradient Surgery for Multi-Task Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight MergingMinsik Choi, Geewook KimICML 2026 · 1 citation
  2. Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMsZhenyu Liu, Xuanyu Zhang, Yunxin Li, Qixun Teng et al.ACL 2026

Builds on16

  1. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  2. Gradient Surgery for Multi-Task LearningTianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine et al.NeurIPS 2020 · 2,261 citations
  3. GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
  4. Scaling Vision with Sparse Mixture of ExpertsCarlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann et al.NeurIPS 2021 · 1,213 citations
  5. Conflict-Averse Gradient Descent for Multi-task learningBo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone et al.NeurIPS 2021 · 686 citations

Related papers

  1. Unveiling Multimodal Processing: Exploring Activation Patterns in Multimodal LLMs for Interpretability and EfficiencyChuan Wu, Meng Su, Youxuan Fang, Shaolin ZhuEMNLP Findings 2025
  2. SD-MoE: Spectral Decomposition for Effective Expert SpecializationRuijun Huang, Fang DONG(董方), Xin Zhang, Anrui Chen et al.ICML 2026 · 2 citations
  3. The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert LevelJeremy Herbst, Stefan Wermter, Jae Hee LeeICML 2026 · 9 citations
  4. UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity Mixture-of-ExpertsZhenyu Liu, Yunxin Li, Xuanyu Zhang, Qixun Teng et al.ACL 2026
  5. Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio EncodersWeiqiao Shan, Yuang Li, Yuhao Zhang, Yingfeng Luo et al.EMNLP 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines