EMNLP Findings2024
Pruning Foundation Models for High Accuracy without Retraining
Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, Xue Lin
Abstract
Despite the superior performance, it is challenging to deploy foundation models or large language models (LLMs) due to their massive parameters and computations. While pruning is a promising technique to reduce model size and accelerate the inference, the traditional pruning techniques can hardly be applied for LLMs as they need to finetune the model on the full dataset with multiple epochs consuming massive data and hardware resources. To deal with this problem, post-training pruning methods are proposed to prune LLMs in one-shot without retraining. However, their accuracy after pruning may suffer from certain performance degradation due to the lack of retraining with massive data. To address this issue, in this paper, we first formulate the post-training problem for layer-wise LLM compression to simultaneously prune multiple weights in LLMs. Next, we provide an optimal solution for this problem and design our post-training pruning algorithm for both unstructured and semi-structured sparsity. Our extensive experiments demonstrate the superior performance of the proposed methods in comparison to SOTA baselines across various LLM families including transformerbased LLMs and Mamba-based LLMs.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- A Simple and Effective Pruning Approach for Large Language Models cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- A Simple and Effective Pruning Approach for Large Language Models cites SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
- Search for Efficient Large Language Models cites A Simple and Effective Pruning Approach for Large Language Models
- Search for Efficient Large Language Models cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Search for Efficient Large Language Models cites SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers cites Up or Down? Adaptive Rounding for Post-Training Quantization
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment cites A Simple and Effective Pruning Approach for Large Language Models
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment cites Search for Efficient Large Language Models
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment cites SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
- Efficient Reasoning with Hidden Thinking cites Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment
- Efficient Reasoning with Hidden Thinking cites LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers
- LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers cites Search for Efficient Large Language Models
- Fast and Memory-Efficient Video Diffusion Using Streamlined Inference cites A Simple and Effective Pruning Approach for Large Language Models
- Fast and Memory-Efficient Video Diffusion Using Streamlined Inference cites SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot cites Up or Down? Adaptive Rounding for Post-Training Quantization
Cited by top-tier papers11
- Efficient Reasoning with Hidden ThinkingXuan Shen, Yizhou Wang, Yufa Zhou, Xiangxi Shi et al.ICML 2026 · 56 citations
- LazyDiT: Lazy Learning for the Acceleration of Diffusion TransformersXuan Shen, Zhao Song, Yufa Zhou, Bo Chen et al.AAAI 2025 · 40 citations
- Search for Efficient Large Language ModelsXuan Shen, Pu Zhao, Yifan Gong, Zhenglun Kong et al.NeurIPS 2024 · 23 citations
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance AssessmentJun Liu, Zhenglun Kong, Pu Zhao, Changdi Yang et al.AAAI 2025 · 23 citations
- Fast and Memory-Efficient Video Diffusion Using Streamlined InferenceZheng Zhan, Yushu Wu, Yifan Gong, Zichong Meng et al.NeurIPS 2024 · 23 citations
Builds on17
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotElias Frantar, Dan AlistarhICML 2023 · 1,240 citations
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale TransformersZhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu et al.NeurIPS 2022 · 816 citations
- Up or Down? Adaptive Rounding for Post-Training QuantizationMarkus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos et al.ICML 2020 · 816 citations
- A Simple and Effective Pruning Approach for Large Language ModelsMingjie Sun, Zhuang Liu, Anna Bair, J. Zico KolterICLR 2024 · 794 citations
Related papers
- ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix FactorizationLawrence Liu, Alexander Liu, Mengdi Wang, Tuo Zhao et al.ICLR 2026 · 3 citations
- LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao WangNeurIPS 2023 · 994 citations
- Fluctuation-Based Adaptive Structured Pruning for Large Language ModelsYongqi An, Xu Zhao, Tao Yu, Ming Tang et al.AAAI 2024 · 130 citations
- Learning Semi-Structured Sparsity for LLMs via Shared and Context-Aware HypernetworkLu Sun, Jun SakumaICLR 2026
- Structured Optimal Brain Pruning for Large Language ModelsJiateng Wei, Quan Lu, Ning Jiang, Siqi Li et al.EMNLP 2024 · 2 citations
