Lune

ACL Findings2025

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts

Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, Jianfeng Gao

Abstract

In this work, we address the memory overhead of deploying Mixture-of-Experts (MoE) architectures in Large Language Models (LLMs). While MoE layers improve LLM performance without increasing inference costs, the evergrowing number of experts inflates memory requirements, hindering practical deployment. Our empirical study reveals that some experts encode redundant knowledge during pretraining. We thus propose a method of grouping and pruning similar experts to improve the model's parameter efficiency. We validate the effectiveness of our method by pruning three state-of-the-art MoE architectures, including Mixtral, Deepseek-MoE, and Qwen. The evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

10 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning cites Measuring Massive Multitask Language Understanding
  • DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  • Unveiling Super Experts in Mixture-of-Experts Large Language Models cites Measuring Massive Multitask Language Understanding
  • Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning cites Measuring Massive Multitask Language Understanding
  • Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource cites Measuring Massive Multitask Language Understanding
  • Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  • Discovering Important Experts for Mixture-of-Experts Models Pruning Through a Theoretical Perspective cites Measuring Massive Multitask Language Understanding
  • Discovering Important Experts for Mixture-of-Experts Models Pruning Through a Theoretical Perspective cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  • MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs cites Measuring Massive Multitask Language Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers10

  1. DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert PruningSikai Bai, Haoxi Li, Jie Zhang, Zicong Hong et al.NeurIPS 2025 · 27 citations
  2. Unveiling Super Experts in Mixture-of-Experts Large Language ModelsZunhai Su, Qingyuan Li, HaoZhang, Weihao Ye et al.ICLR 2026 · 16 citations
  3. Discovering Important Experts for Mixture-of-Experts Models Pruning Through a Theoretical PerspectiveWeizhong Huang, Yuxin Zhang, Xiawu Zheng, Fei Chao et al.NeurIPS 2025 · 12 citations
  4. MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMsXiaodong Chen, Mingming Ha, Zhenzhong Lan, Jing Zhang et al.ICLR 2026 · 12 citations
  5. Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal ResourceHouyi Li, Ka Man Lo, Shijie Xuyang, Ziqi Wang et al.ICLR 2026 · 8 citations

Builds on19

  1. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  2. GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
  3. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs et al.ICML 2022 · 1,464 citations
  4. Sheared LLaMA: Accelerating Language Model Pre-training via Structured PruningMengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi ChenICLR 2024 · 453 citations
  5. Pruning from ScratchYulong Wang, Xiaolu Zhang, Lingxi Xie, Jun Zhou et al.AAAI 2020 · 219 citations

Related papers

  1. Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMsYixiao Zhou, Ziyu Zhao, Dongzhou Cheng, Zhiliang Wu et al.EMNLP Findings 2025 · 4 citations
  2. MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value DecompositionWei Li, Lujun Li, Hao Gu, You-Liang Huang et al.ICML 2025
  3. COMPEL: Compensated Mixture-of-Experts Pruning with Expert-Layer distributionSeohee Yoon, Yong Suk ChoiACL Findings 2026
  4. Retraining-free Merging of Sparse MoE via Hierarchical ClusteringI-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao et al.ICML 2025
  5. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language ModelsXudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou et al.ACL 2024 · 16 citations
Dusk over the sea between two cliffs drawn in fine vertical lines