ACL Findings2023
Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation
Maha Elbayad, Anna Y. Sun, Shruti Bhosale
Abstract
Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation. However, for lowresource tasks, MoE models severely over-fit. We introduce in this work effective regularization strategies, namely (1) dropout techniques for MoE layers in Expert Output Masking (EOM) and Final Output Masking (FOM), (2) Conditional MoE Routing (CMR) that learns what tokens require the extra capacity of MoE layers and (3) Curriculum Learning methods that introduce low-resource pairs at later stages of training. All these methods prevent over-fitting and improve the performance of MoE models on low-resource tasks without adversely affecting high-resource tasks. On a massively multilingual machine translation benchmark, our strategies result in about +1 chrF ++ improvement in very low resource language pairs.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Sparse MoE with Language Guided Routing for Multilingual Machine Translation cites Language Models are Few-Shot Learners
- Sparse MoE with Language Guided Routing for Multilingual Machine Translation cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Sparse MoE with Language Guided Routing for Multilingual Machine Translation cites DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
- Mixture of Languages: Improved Multilingual Encoders Through Language Grouping cites Sparse MoE with Language Guided Routing for Multilingual Machine Translation
- Mixture of Languages: Improved Multilingual Encoders Through Language Grouping cites GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- Mixture of Languages: Improved Multilingual Encoders Through Language Grouping cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Integrating Plutchik's Theory with Mixture of Experts for Enhancing Emotion Classification cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models cites GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding cites Language Models are Few-Shot Learners
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale cites GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale cites Language Models are Few-Shot Learners
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Cited by top-tier papers5
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDamai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu et al.ACL 2024 · 171 citations
- Sparse MoE with Language Guided Routing for Multilingual Machine TranslationXinyu Zhao, Xuxi Chen, Yu Cheng, Tianlong ChenICLR 2024 · 19 citations
- Integrating Plutchik's Theory with Mixture of Experts for Enhancing Emotion ClassificationDongjun Lim, Yun-Gyung CheongEMNLP 2024 · 4 citations
- THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine TranslationYunlong Liang, Fandong Meng, Jie ZhouACL 2025 · 1 citation
- Mixture of Languages: Improved Multilingual Encoders Through Language GroupingJoão Maria Janeiro, Belen Alastruey, Francisco Massa, Maha Elbayad et al.EMNLP 2025
Builds on14
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
- On Layer Normalization in the Transformer ArchitectureRuibin Xiong, Yunchang Yang, Di He, Kai Zheng et al.ICML 2020 · 1,388 citations
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI ScaleSamyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang et al.ICML 2022 · 523 citations
Related papers
- MoEC: Mixture of Expert ClustersYuan Xie, Shaohan Huang, Tianyu Chen, Furu WeiAAAI 2023 · 27 citations
- MMNMT: Modularizing Multilingual Neural Machine Translation with Flexibly Assembled MoE and Dense BlocksShangjie Li, Xiangpeng Wei, Shaolin Zhu, Jun Xie et al.EMNLP 2023 · 4 citations
- Gating Dropout: Communication-efficient Regularization for Sparsely Activated TransformersRui Liu, Young Jin Kim, Alexandre Muzio, Hany HassanICML 2022 · 31 citations
- Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic CapacityHaoran Xu, Maha Elbayad, Kenton Murray, Jean Maillard et al.EMNLP Findings 2023 · 4 citations
- Beyond Distillation: Task-level Mixture-of-Experts for Efficient InferenceSneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun et al.EMNLP Findings 2021 · 41 citations
