Mixtures of Experts for Audio-Visual Learning
Ying Cheng, Yang Li, Junjie He, Rui Feng
Abstract
With the rapid development of multimedia technology, audio-visual learning has emerged as a promising research topic within the field of multimodal analysis. In this paper, we explore parameter-efficient transfer learning for audio-visual learning and propose the Audio-Visual Mixture of Experts (AVMoE) to inject adapters into pre-trained models flexibly. Specifically, we introduce unimodal and cross-modal adapters as multiple experts to specialize in intra-modal and intermodal information, respectively, and employ a lightweight router to dynamically allocate the weights of each expert according to the specific demands of each task. Extensive experiments demonstrate that our proposed approach AVMoE achieves superior performance across multiple audio-visual tasks, including AVE, AVVP, AVS, and AVQA. Furthermore, visual-only experimental results also indicate that our approach can tackle challenging scenes where modality information is missing.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Scaling Vision with Sparse Mixture of Experts cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition cites GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition cites MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition
- MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Swin Transformer V2: Scaling Up Capacity and Resolution cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- Swin Transformer V2: Scaling Up Capacity and Resolution cites Scaling Vision with Sparse Mixture of Experts
Cited by top-tier papers4
- Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided GateLiangwei Zheng, Wei Emma Zhang, Mingyu Guo, Olaf Maennel et al.ICML 2026 · 7 citations
- MoME: Mixture of Matryoshka Experts for Audio-Visual Speech RecognitionUmberto Cappellazzo, Minsu Kim, Pingchuan Ma, Honglie Chen et al.NeurIPS 2025 · 5 citations
- Exposing and Defending the Achilles' Heel of Video Mixture-of-ExpertsSongping Wang, Qinglong Liu, Yueming Lyu, Ning Li et al.ICLR 2026 · 3 citations
- MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech RecognitionSungnyun Kim, Kangwook Jang, Sangmin Bae, Sungwoo Cho et al.ICML 2025
Builds on32
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
- Swin Transformer V2: Scaling Up Capacity and ResolutionZe Liu, Han Hu, Yutong Lin, Zhuliang Yao et al.CVPR 2022 · 2,138 citations
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
- Scaling Vision with Sparse Mixture of ExpertsCarlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann et al.NeurIPS 2021 · 1,213 citations
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
Related papers
- pMoE: Prompting Diverse Experts Together Wins More in Visual AdaptationShentong Mo, Xufang Luo, Dongsheng LiICLR 2025
- π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task InterpolationChengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu et al.ICML 2023 · 50 citations
- PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer LearningZhisheng Lin, Han Fu, Chenghao Liu, Zhuo Li et al.ACL Findings 2024 · 2 citations
- Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained ExpertsYangyang Xu, Xi Ye, Duo SuACM MM 2025
- LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task LearningMd Kowsher, Haris Mansoor, Nusrat Prottasha, Ozlem Garibay et al.ICML 2026
