Lune

ACL Findings2026

SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment

Tianyu Dong, Yangyang Liu, Jiang Zhou, Xinwei Wu, Xiaohu Zhao, Hao Wang, Heng Liu, Linlong Xu, Longyue Wang, Weihua Luo, Shaolin Zhu, Deyi Xiong

Abstract

Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational efficiency. However, low-resource languages-which suffer from a scarcity of highquality training data-often have their tokens routed to different experts than those predominantly activated by high-resource inputs, which limits cross-lingual expert sharing. This crosslingual routing divergence consequently hinders their efficacy in multilingual contexts. To address this issue, we propose SARA (Semantically Anchored Routing Alignment), a framework designed to transfer specialized capabilities from high-resource languages as anchors to low-resource languages. SARA explicitly aligns the routing distribution of multilingual inputs with high-resource semantic anchors using a symmetric Jensen-Shannon (JS) divergence constraint. Unlike traditional distillation methods that operate on output logits, SARA directly aligns the internal routing distributions of MoE layers, encouraging mechanistic consistency in expert selection across languages. We conduct experiments on 2 LLMs across 5 low-resource languages and 3 benchmarks. Experiment results demonstrate that SARA outperforms standard instruction tuning (e.g., +0.8% on Qwen3-30B-A3B and +1.2% on Phi-3.5-MoE-instruct on Global-MMLU benchmark). Further analyses show that SARA effectively addresses performance bottlenecks in low-resource languages, providing a scalable pathway to enhance multilingual capabilities in sparse architectures. Our code is available at https://github.com/iMoriton/sara.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Multilingual Routing in Mixture-of-Experts cites The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
  • Multilingual Routing in Mixture-of-Experts cites Language models are multilingual chain-of-thought reasoners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on22

  1. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations
  2. Language models are multilingual chain-of-thought reasonersFreda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang et al.ICLR 2023 · 52 citations
  3. Advancing Expert Specialization for Better MoEHongcan Guo, Haolang Lu, Guoshun Nan, Bolun Chu et al.NeurIPS 2025 · 48 citations
  4. Multilingual Routing in Mixture-of-ExpertsLucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu et al.ICLR 2026 · 34 citations
  5. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language VariantsLucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe et al.ACL 2024 · 30 citations

Related papers

  1. Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family ExpertsGuorui Zheng, Xidong Wang, Juhao Liang, Nuo Chen et al.ICLR 2025
  2. Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-ExpertsShengzhuang Chen, Ying Wei, Jonathan Richard SchwarzACL 2025
  3. Hybrid Routing for a Mixture of LoRA ExpertsYitong Huang, Ziqi Yang, Zihui Wang, Jianzhong Qi et al.AAAI 2026
  4. Expert Divergence Learning for MoE-based Language ModelsJiaang Li, Haibin Chen, Langming Liu, Yujin Yuan et al.ICLR 2026 · 3 citations
  5. A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMΔ Integration into Upcycled MoEHao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She et al.ACL 2026
Dusk over the sea between two cliffs drawn in fine vertical lines