EMNLP Findings2025
BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration
Abstract
Mixture-of-Experts (MoE) models have become increasingly powerful in multimodal learning by enabling modular specialization across modalities. However, their effectiveness remains unclear when additional modalities introduce more noise than complementary information. Existing approaches, like the Partial Information Decomposition, struggle to scale beyond two modalities and lack instance-level control. We propose Beyond Two-modality Weighting (BTW) 1 , a bi-level, non-parametric weighting framework that combines instancelevel Kullback-Leibler (KL) divergence and modality-level mutual information (MI) to dynamically adjust modality importance during training. Our method requires no extra parameters and supports any number of modalities. Specifically, BTW computes per-example KL weights by measuring divergence between each unimodal and the current multimodal prediction, and modality-wide MI weights by estimating global alignment between unimodal and multimodal outputs. Extensive experiments on sentiment regression and clinical classification demonstrate that our method significantly improves regression performance and multiclass classification accuracy.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts cites Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts cites Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusion
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts cites FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
- Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusion cites Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
- FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion cites Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
- FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion cites Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusion
Builds on13
- Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of ExpertsBasil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton et al.NeurIPS 2022 · 359 citations
- FuseMoE: Mixture-of-Experts Transformers for Fleximodal FusionXing Han, Huy Nguyen, Carl Harris, Nhat Ho et al.NeurIPS 2024 · 129 citations
- Quantifying & Modeling Multimodal Interactions: An Information Decomposition FrameworkPaul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling et al.NeurIPS 2023 · 120 citations
- Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image FusionBing Cao, Yiming Sun, Pengfei Zhu, Qinghua HuICCV 2023 · 110 citations
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-ExpertsSukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu et al.NeurIPS 2024 · 98 citations
Related papers
- I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-ExpertsJiayi Xin, Sukwon Yun, Jie Peng, Inyoung Choi et al.ICML 2025
- EMOE: Modality-Specific Enhanced Dynamic Emotion ExpertsYiyang Fang, Wenke Huang, Guancheng Wan, Kehua Su et al.CVPR 2025
- CP-IMoE: Collaborative Prompt-Guided Interactive Mixture-of-Experts for Incomplete Multimodal LearningJing Li, Dongbo Zhang, Yalin Zheng, Yanda MengCVPR Findings 2026
- Leveraging Knowledge of Modality Experts for Incomplete Multimodal LearningWenxin Xu, Hexin Jiang, Xuefeng LiangACM MM 2024 · 31 citations
- R²MoE: Representation and Expert Selection Dual-Regularized Mixture-of-Experts for Multimodal Clinical DataWajih Hassan Raza, Mya Schiess, Juan Martinez Lemus, Timothy Michael Ellmore, Charles Green, Claudio Soto, Xin Fu, Renjie HuCVPR Findings 2026
