EMNLP Findings2023
Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity
Haoran Xu, Maha Elbayad, Kenton Murray, Jean Maillard, Vedanuj Goswami
Abstract
Mixture-of-experts (MoE) models that employ sparse activation have demonstrated effectiveness in significantly increasing the number of parameters while maintaining low computational requirements per token. However, recent studies (Hoffmann et al., 2022;Zuo et al., 2021;Gao et al., 2022) have established that MoE models are inherently parameter-inefficient as the improvement in performance diminishes with an increasing number of experts. We hypothesize this parameter inefficiency is a result of all experts having equal capacity, which may not adequately meet the varying complexity requirements of different tokens or tasks. In light of this, we propose Stratified Mixture of Experts (SMoE) models, which feature a stratified structure and can assign dynamic capacity to different tokens. We demonstrate the effectiveness of SMoE on three multilingual machine translation benchmarks, containing 4, 15, and 94 language pairs, respectively. We show that SMoE outperforms multiple state-of-the-art MoE models with the same or fewer parameters. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- An empirical analysis of compute-optimal large language model training cites GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- Condensing Multilingual Knowledge with Lightweight Language-Specific Modules cites An empirical analysis of compute-optimal large language model training
- Condensing Multilingual Knowledge with Lightweight Language-Specific Modules cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Mixture-of-Experts with Expert Choice Routing cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Cited by top-tier papers1
Builds on12
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
- On Layer Normalization in the Transformer ArchitectureRuibin Xiong, Yunchang Yang, Di He, Kai Zheng et al.ICML 2020 · 1,388 citations
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
- Mixture-of-Experts with Expert Choice RoutingYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du et al.NeurIPS 2022 · 933 citations
- An empirical analysis of compute-optimal large language model trainingJordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya et al.NeurIPS 2022 · 566 citations
Related papers
- XMoE: Sparse Models with Fine-grained and Adaptive Expert SelectionYuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang et al.ACL Findings 2024 · 10 citations
- HMoE: Heterogeneous Mixture of Experts for Language ModelingAn Wang, Xingwu Sun, Ruobing Xie, Shuaipeng Li et al.EMNLP 2025 · 2 citations
- Multi-Head Mixture-of-ExpertsXun Wu, Shaohan Huang, Wenhui Wang, Shuming Ma et al.NeurIPS 2024 · 42 citations
- Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of ExpertsShwai He, Weilin Cai, Jiayi Huang, Ang LiICLR 2026 · 18 citations
- Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer ModelsYongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu et al.ICLR 2025
