The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models
Yan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang, Zining Zhu
摘要
Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing. In this work, we question this assumption by introducing COMMITTEEAUDIT, a post hoc framework that analyzes routing behavior at the level of expert groups rather than individual experts. Across three representative models and the MMLU benchmark, we uncover a domain invariant Standing Committee. This is a compact coalition of routed experts that consistently captures the majority of routing mass across domains, layers, and routing budgets, even when architectures already include shared experts. Qualitative analysis further shows that Standing Committees anchor reasoning structure and syntax, while peripheral experts handle domain-specific knowledge. These findings reveal a strong structural bias toward centralized computation, suggesting that specialization in Mixture of Experts models is far less pervasive than commonly believed. Crucially, this inherent bias indicates that current training objectives, such as load-balancing losses that enforce uniform expert utilization, may be working against the model's natural optimization path, thereby limiting training efficiency and performance. The code is available at GitHub 1 .
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper3
- OpenMoE: An Early Effort on Open Mixture-of-Experts Language ModelsFuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni 等ICML 2024 · 被引用 183 次
- Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family ExpertsGuorui Zheng, Xidong Wang, Juhao Liang, Nuo Chen 等ICLR 2025
- Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMsJun Bai, Minghao Tong, Yang Liu, Zixia Jia 等EMNLP 2025
相关 Paper
- Hierarchical Mixture of Experts with Two-Stage OptimizationGleb Molodtsov, Alexander Miasnikov, Aleksandr BeznosikovKDD 2026 · 被引用 2 次
- The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert LevelJeremy Herbst, Stefan Wermter, Jae Hee LeeICML 2026 · 被引用 9 次
- Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of ExpertsYongXiang Hua, Haoyu Cao, Zhou Tao, Bocheng Li 等ACM MM 2025 · 被引用 1 次
- STAR: Rethinking MoE Routing as Structure-Aware Subspace LearningSumin Park, Noseong ParkICML 2026
- Uncertainty-Aware Routing for Principled Alignment with MoE DynamicsYilong Chen, Junyuan Shang, Yuchen Feng, Zhenyu Zhang 等ACL 2026
