Lune

ACL Findings2025

Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model

Chong Li, Yingzhuo Deng, Jiajun Zhang, Chengqing Zong

Abstract

The curse of multilinguality phenomenon is a fundamental problem of multilingual Large Language Models (LLMs), where the competition between massive languages results in inferior performance.It mainly comes from limited capacity and negative transfer between dissimilar languages.To address this issue, we propose a method to dynamically group and scale up the parameters of multilingual LLM while boosting positive transfer among similar languages.Specifically, the model is first tuned on monolingual corpus to determine the parameter deviation in each layer and quantify the similarity between languages.Layers with more deviations are extended to mixture-of-experts layers to reduce competition between languages, where one expert module serves one group of similar languages.Experimental results on 18 to 128 languages show that our method reduces the negative transfer between languages and significantly boosts multilingual performance with fewer parameters.Such language group specialization on experts benefits the new language adaptation and reduces the inference on the previous multilingual knowledge learned. 1 1 Our code and model weights are available at https://git hub.com/ZNLP/DMoE ta de ||∆𝜃 𝑙 𝑥 || = ||𝜃 𝑙 0 -𝜃 𝑙 𝑥 || 𝑓𝑜𝑟 𝑙𝑎𝑦𝑒𝑟 𝑙 ∈ [1, 𝑁] 𝒍 5 4 3 2 1 … … id (a) Calculate the parameter deviation for each language.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMΔ Integration into Upcycled MoE cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  • Hash Layers For Large Sparse Models cites Unsupervised Cross-lingual Representation Learning at Scale
  • Hash Layers For Large Sparse Models cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  • DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models cites Hash Layers For Large Sparse Models
  • DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMΔ Integration into Upcycled MoEHao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She et al.ACL 2026
  2. Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit FusionChen Zhang, Jiuheng Lin, Zhiyuan Liao, Yansong FengACL 2026

Builds on15

  1. GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
  2. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  3. Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
  4. Hash Layers For Large Sparse ModelsStephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, Jason WestonNeurIPS 2021 · 316 citations
  5. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDamai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu et al.ACL 2024 · 171 citations

Related papers

  1. Mixture of Languages: Improved Multilingual Encoders Through Language GroupingJoão Maria Janeiro, Belen Alastruey, Francisco Massa, Maha Elbayad et al.EMNLP 2025
  2. Multilingual Routing in Mixture-of-ExpertsLucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu et al.ICLR 2026 · 34 citations
  3. Breaking the Curse of Multilinguality with Cross-lingual Expert Language ModelsTerra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li et al.EMNLP 2024 · 6 citations
  4. Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-ExpertsXue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang et al.ACL 2025 · 9 citations
  5. On Negative Interference in Multilingual Models: Findings and A Meta-Learning TreatmentZirui Wang, Zachary C. Lipton, Yulia TsvetkovEMNLP 2020 · 72 citations
Dusk over the sea between two cliffs drawn in fine vertical lines