Lune

ACL Findings2026

From Outcome to Process: Optimizing MoE Load Balancing with MCTS

Wenjun Ke, Hengyuan Xu, Ziyu Shang, Yao He, Jiahao Wang, Zijie Xu, Peng Wang, Yuhang Lou, Jiajun Liu

Abstract

Mixture of Experts (MoE) dynamically routes inputs to specialized expert networks, enabling large language models to scale capacity with low inference overhead. To further improve MoE's parameter efficiency in resourceconstrained scenarios, LoRA-MoE integrates LoRA for lightweight adaptation while preserving MoE's specialization. Despite these benefits, the effectiveness of LoRA-MoE still hinges on balanced expert utilization, where certain experts dominate activations while most remain underutilized. Existing balancing strategies focus on constraining the final distribution of expert usage, but overlook the routing decisions made at each layer. As a result, imbalances gradually accumulate across the routing hierarchy. To address this challenge, we propose LayerMoE, a novel threestage framework that leverages process-level rewards to guide balanced expert routing. Specifically, to overcome the limitation of focusing only on final losses and ignoring intermediate routing, we introduce Monte Carlo Tree Search (MCTS)-based sampling that decomposes outcome-level supervision into layerwise reward signals, guiding expert choices throughout the routing process. For efficiency, we organize Transformer layers into groups, which constrain the search space of MCTS and keep exploration overhead tractable while retaining the hierarchical structure. Extensive experiments on representative datasets (e.g., ARC, RACE, OBQA) show that applying LayerMoE consistently improves the performance of stateof-the-art LoRA-MoE baselines, yielding an average accuracy gain of 1.39%. Notably, the maximum improvement reaches 2.50%.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Language Models as Commonsense Knowledge for Large-Scale Task Planning cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
  • SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on17

  1. Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
  2. GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
  3. Large Language Models as Commonsense Knowledge for Large-Scale Task PlanningZirui Zhao, Wee Sun Lee, David HsuNeurIPS 2023 · 423 citations
  4. HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-TuningChunlin Tian, Zhan Shi, Zhijiang Guo, Li Li et al.NeurIPS 2024 · 172 citations
  5. SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-ImprovementXiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu et al.NeurIPS 2025 · 158 citations

Related papers

  1. Hybrid Routing for a Mixture of LoRA ExpertsYitong Huang, Ziqi Yang, Zihui Wang, Jianzhong Qi et al.AAAI 2026
  2. LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention RoutingWenbing Li, Zikai Song, Hang Zhou, Junqing Yu et al.ICLR 2026 · 20 citations
  3. AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language ModelsZihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang et al.EMNLP Findings 2024 · 8 citations
  4. TC-MoE: Augmenting Mixture of Experts with Ternary Expert ChoiceShen Yan, Xingyan Bin, Sijun Zhang, Yisen Wang et al.ICLR 2025
  5. HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA ExpertsMengqi Liao, Wei Chen, Junfeng Shen, Shengnan Guo et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines