EMNLP Findings2025
Exploration-Driven Reinforcement Learning for Expert Routing Improvement in Mixture-of-Experts Language Models
Gyunyeop Kim, Sangwoo Kang
Abstract
The performance of MoE-based LLMs depends on the router's ability to select suitable experts; however, the router is typically not explicitly supervised to acquire this routing ability. We propose Exploration-Driven Reinforcement Learning (ERL), which explicitly optimizes the router by exploration of alternative routing paths. For every input, ERL evaluates by (i) the original routing path and (ii) paths in which an α-fraction of routing decisions is randomly perturbed, and treats their performance gap as an advantage signal in a reinforcement learning. Moreover, MoE-ERL wP L mitigates the risk of performance collapse caused by routing reinforcement learning-induced expert overspecialization by intentionally enforcing overlap in experts' knowledge. Without adding parameters or external reward models, our method improves summarization (SAMSum, XSUM), question answering (SQuAD), and language modeling (WikiText-2), and raises routing quality, delivering up to 8.9 × higher MRR than baselines over 100 perturbed routing paths. Code is available at our github 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models cites Hash Layers For Large Sparse Models
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models cites Mixture-of-Experts with Expert Choice Routing
- Mixture-of-Experts with Expert Choice Routing cites Hash Layers For Large Sparse Models
- Toward Efficient Inference for Mixture of Experts cites Hash Layers For Large Sparse Models
- Toward Efficient Inference for Mixture of Experts cites DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Builds on12
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Mixture-of-Experts with Expert Choice RoutingYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du et al.NeurIPS 2022 · 933 citations
- Hash Layers For Large Sparse ModelsStephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, Jason WestonNeurIPS 2021 · 316 citations
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDamai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu et al.ACL 2024 · 171 citations
- Toward Efficient Inference for Mixture of ExpertsHaiyang Huang, Newsha Ardalani, Anna Y. Sun, Liu Ke et al.NeurIPS 2024 · 60 citations
Related papers
- MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language ModelsDohwan Ko, Jinyoung Park, Seoung Choi, Sanghyeok Lee et al.CVPR 2026 · 3 citations
- Improved Policy Optimization for Mixture-of-Experts Models: Importance Sampling and Rewarding from an Expert-Centric PerspectiveYining Qian, Jinpeng Li, Fei Mi, Lifeng Shang et al.ACL Findings 2026
- Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement LearningHaozhen Zhang, Tao Feng, Jiaxuan YouNeurIPS 2025 · 81 citations
- Self-MoE: Towards Compositional Large Language Models with Self-Specialized ExpertsJunmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang et al.ICLR 2025
- Multilingual Routing in Mixture-of-ExpertsLucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu et al.ICLR 2026 · 34 citations
