ACL Findings2026
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
摘要
Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts. Existing compression approaches either remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency. To address the limitations, we propose TENP, a structured Trapezoidal Expert Neuron Pruning framework. Using a few samples, we identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts, preserving model parameters in a trapezoidal pattern from shallow to deep layers. When evaluating expert importance, we jointly consider both the magnitude of the expert output and its ability to change the direction of the input vector. For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons. We conduct extensive experiments on the Qwen and DeepSeek models. Under a routing expert sparsity of 40% and an average of 63.76% activated expert parameters, the DeepSeek model suffers only a 1-point drop in accuracy compared to the full-parameter model. Moreover, it outperforms the full-parameter model by 10% on code generation tasks.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
脉络
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy 引用了 GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- A Simple and Effective Pruning Approach for Large Language Models 引用了 SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models 引用了 GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
它引用的 Paper13
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen 等ICLR 2021 · 1,954 次引用
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotElias Frantar, Dan AlistarhICML 2023 · 1,240 次引用
- A Simple and Effective Pruning Approach for Large Language ModelsMingjie Sun, Zhuang Liu, Anna Bair, J. Zico KolterICLR 2024 · 794 次引用
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDamai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu 等ACL 2024 · 171 次引用
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing PolicyPingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung 等ICLR 2024 · 97 次引用
相关 Paper
- Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMsYixiao Zhou, Ziyu Zhao, Dongzhou Cheng, Zhiliang Wu 等EMNLP Findings 2025 · 4 次引用
- C-GNN-PRUNE: A Unified Graph-Based Framework for Structure-Aware Pruning of Mixture-of-Experts ModelsLin Li, Yan Wang, Zhuopeng WangAAAI 2026 · 1 次引用
- Attribution-Guided and Coverage-Maximized Pruning for Structural MoE CompressionYifu Ding, jiacheng wang, Ge Yang, Yongcheng Jing 等ICML 2026
- Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-ExpertsZeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu 等ACL Findings 2025 · 5 次引用
- Less Token, More Signal: MoE Expert Pruning via Critical Token SelectionZeliang Zong, Kai Zhang, Yarong Wang, wenming tan 等ICML 2026
