EMNLP Findings2025
Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs
Yixiao Zhou, Ziyu Zhao, Dongzhou Cheng, Zhiliang Wu, Jie Gui, Yi Yang, Fei Wu, Yu Cheng, Hehe Fan
Abstract
Sparse Mixture-of-Experts (SMoE) architectures are widely used in large language models (LLMs) due to their computational efficiency.However, though only a few experts are activated for each token, SMoE still requires loading all expert parameters, leading to high memory usage and challenges in deployment.Previous work has tried to reduce the overhead by pruning and merging experts, but primarily focused on expert-level operations, leaving neuron-level structure underexplored.We propose DERN (Dropping Experts, Recombining Neurons), a task-agnostic and retraining-free framework for expert pruning and reconstruction.We observe that experts are often misaligned and contain semantic conflicts at the neuron level, which poses challenges for direct merging.To solve this, DERN works in three steps: it first prunes redundant experts using router statistics; then it decomposes them into neuron-level expert segments, assigning each segment to its most compatible retained expert; and finally, it merges segments within each retained expert to build a compact representation.Experiments on Mixtral, Qwen, and DeepSeek SMoE models show that DERN achieves over a 5% performance gains than previous methods on commonsense reasoning and MMLU benchmarks under 50% expert sparsity, without extra training.It also greatly reduces the number of experts and memory usage, making SMoE LLMs easier to deploy in practice. Table 1: Comparison of our method with other expert pruning methods.E-Merge means expert-level merge, while N-Merge means neuron-level merge.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
13 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- A Simple and Effective Pruning Approach for Large Language Models cites Measuring Massive Multitask Language Understanding
- A Simple and Effective Pruning Approach for Large Language Models cites Language Models are Few-Shot Learners
- Measuring Massive Multitask Language Understanding cites PIQA: Reasoning about Physical Commonsense in Natural Language
- Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
- Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning cites SepPrune: Structured Pruning for Efficient Deep Speech Separation
- Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning cites Language Models are Few-Shot Learners
- SepPrune: Structured Pruning for Efficient Deep Speech Separation cites A Simple and Effective Pruning Approach for Large Language Models
- Efficient Memory Management for Large Language Model Serving with PagedAttention cites Language Models are Few-Shot Learners
Cited by top-tier papers13
- SepPrune: Structured Pruning for Efficient Deep Speech SeparationYuqi Li, Kai Li, Xin Yin, Zhifei Yang et al.AAAI 2026 · 4 citations
- FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual LearningYujie Feng, Hao Wang, Jian Li, Xu Chu et al.ACL 2026 · 3 citations
- Rethinking Table Pruning in TableQA: From Sequential Revisions to Gold Trajectory-Supervised Parallel SearchYu Guo, Shenghao Ye, Shuangwu Chen, Zijian Wen et al.ACL 2026 · 2 citations
- Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-TuningHua Ye, Siyuan Chen, Haoliang Zhang, Weihao Luo et al.NeurIPS 2025 · 2 citations
- Whose Instructions Count? Resolving Preference Bias in Instruction Fine-TuningJiayu Zhang, Changbang Li, Yinan Peng, Weihao Luo et al.NeurIPS 2025 · 2 citations
Builds on18
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
- A Simple and Effective Pruning Approach for Large Language ModelsMingjie Sun, Zhuang Liu, Anna Bair, J. Zico KolterICLR 2024 · 794 citations
Related papers
- Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-ExpertsZeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu et al.ACL Findings 2025 · 5 citations
- Retraining-free Merging of Sparse MoE via Hierarchical ClusteringI-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao et al.ICML 2025
- REAP the Experts: Why Pruning Prevails for One-Shot MoE compressionMike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie et al.ICLR 2026 · 47 citations
- Less Token, More Signal: MoE Expert Pruning via Critical Token SelectionZeliang Zong, Kai Zhang, Yarong Wang, wenming tan et al.ICML 2026
- Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts ModelsWentao Hu, Mingkuan Zhao, Shuangyong Song, Xiaoyan Zhu et al.AAAI 2026 · 4 citations
