Lune

EMNLP Findings2024

AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models

Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, Zhijie Deng

Abstract

Mixture of experts (MoE) has become the standard for constructing production-level large language models (LLMs) due to its promise to boost model capacity without causing significant overheads.Nevertheless, existing MoE methods usually enforce a constant top-k routing for all tokens, which is arguably restrictive because various tokens (e.g., "<EOS>" vs. "apple") may require various numbers of experts for feature abstraction.Lifting such a constraint can help make the most of limited resources and unleash the potential of the model for downstream tasks.In this sense, we introduce AdaMOE to realize token-adaptive routing for MoE, where different tokens are permitted to select a various number of experts.AdaMOE makes minimal modifications to the vanilla MoE with top-k routing-it simply introduces a fixed number of null experts, which do not consume any FLOPs, to the expert set and increases the value of k.AdaMOE does not force each token to occupy a fixed number of null experts but ensures the average usage of the null experts with a load-balancing loss, leading to an adaptive number of null/true experts used by each token.AdaMOE exhibits a strong resemblance to MoEs with expert choice routing while allowing for trivial auto-regressive modeling.AdaMOE is easy to implement and can be effectively applied to pre-trained (MoE-)LLMs.Extensive studies show that AdaMOE can reduce average expert load (FLOPs) while achieving superior performance.For example, on the ARC-C dataset, applying our method to fine-tuning Mixtral-8x7B can reduce FLOPs by 14.5% while increasing accuracy by 1.69%.Code is available at this link.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

12 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts cites LoRA: Low-Rank Adaptation of Large Language Models
  • QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
  • Advancing Expert Specialization for Better MoE cites QLoRA: Efficient Finetuning of Quantized LLMs
  • SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers12

  1. Advancing Expert Specialization for Better MoEHongcan Guo, Haolang Lu, Guoshun Nan, Bolun Chu et al.NeurIPS 2025 · 48 citations
  2. SonicMoE: Accelerating MoE with IO and Tile-aware OptimizationsWentao Guo, Mayank Mishra, Xinle Cheng, Ion Stoica et al.ICLR 2026 · 22 citations
  3. LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA ExpertsYuan Zhuang, Yi Shen, Yuexin Bian, Qing Su et al.ICLR 2026 · 15 citations
  4. Advancing SMoE for Continuous Domain Adaptation of MLLMs: Adaptive Router and Domain-Specific LossLiang Zhang, Ziyao Lu, Fandong Meng, Hui Li et al.ACL 2025 · 3 citations
  5. Timeexpert: an Expert-Guided Video Llm for Video Temporal GroundingZuhao Yang, Yingchen Yu, Yunqing Zhao, Shijian Lu et al.ICCV 2025 · 3 citations

Builds on14

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  4. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  5. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations

Related papers

  1. Ada-K Routing: Boosting the Efficiency of MoE-based LLMsTongtian Yue, Longteng Guo, Jie Cheng, Xuange Gao et al.ICLR 2025
  2. Autonomy-of-Experts ModelsAng Lv, Ruobing Xie, Yining Qian, Songhao Wu et al.ICML 2025
  3. Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of ExpertsShwai He, Weilin Cai, Jiayi Huang, Ang LiICLR 2026 · 18 citations
  4. Adaptive Gating in Mixture-of-Experts based Language ModelsJiamin Li, Qiang Su, Yitao Yang, Yimin Jiang et al.EMNLP 2023 · 9 citations
  5. Mining Tensor/Neuron-Level Sparsity to Maximize Mixture-of-Experts Potential in Post-Training and InferenceWeilin Cai, Le Qin, Shwai He, Junwei Cui et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines