Lune

ACL Findings2026

TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts

Jiangyang He, Shaolin Zhu, Deyi Xiong

Abstract

Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts. Existing compression approaches either remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency. To address the limitations, we propose TENP, a structured Trapezoidal Expert Neuron Pruning framework. Using a few samples, we identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts, preserving model parameters in a trapezoidal pattern from shallow to deep layers. When evaluating expert importance, we jointly consider both the magnitude of the expert output and its ability to change the direction of the input vector. For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons. We conduct extensive experiments on the Qwen and DeepSeek models. Under a routing expert sparsity of 40% and an average of 63.76% activated expert parameters, the DeepSeek model suffers only a 1-point drop in accuracy compared to the full-parameter model. Moreover, it outperforms the full-parameter model by 10% on code generation tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  • A Simple and Effective Pruning Approach for Large Language Models cites SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
  • DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on13

  1. GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
  2. SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotElias Frantar, Dan AlistarhICML 2023 · 1,240 citations
  3. A Simple and Effective Pruning Approach for Large Language ModelsMingjie Sun, Zhuang Liu, Anna Bair, J. Zico KolterICLR 2024 · 794 citations
  4. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDamai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu et al.ACL 2024 · 171 citations
  5. Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing PolicyPingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung et al.ICLR 2024 · 97 citations

Related papers

  1. Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMsYixiao Zhou, Ziyu Zhao, Dongzhou Cheng, Zhiliang Wu et al.EMNLP Findings 2025 · 4 citations
  2. C-GNN-PRUNE: A Unified Graph-Based Framework for Structure-Aware Pruning of Mixture-of-Experts ModelsLin Li, Yan Wang, Zhuopeng WangAAAI 2026 · 1 citation
  3. Attribution-Guided and Coverage-Maximized Pruning for Structural MoE CompressionYifu Ding, jiacheng wang, Ge Yang, Yongcheng Jing et al.ICML 2026
  4. Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-ExpertsZeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu et al.ACL Findings 2025 · 5 citations
  5. Less Token, More Signal: MoE Expert Pruning via Critical Token SelectionZeliang Zong, Kai Zhang, Yarong Wang, wenming tan et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines