Lune

ACL Findings2024

SP³: Enhancing Structured Pruning via PCA Projection

Yuxuan Hu, Jing Zhang, Zhe Zhao, Chen Zhao, Xiaodong Chen, Cuiping Li, Hong Chen

Abstract

Structured pruning is a widely used technique for reducing the size of pre-trained language models (PLMs), but current methods often overlook the potential of compressing the hidden dimension (d) in PLMs, a dimension critical to model size and efficiency. This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (SP 3 ), targeting the effective reduction of d by projecting features into a space defined by principal components before masking. Extensive experiments on benchmarks (GLUE and SQuAD) show that SP 3 can reduce d by 70%, compress 94% of the BERT base model, and maintain over 96% accuracy and outperform other methods that compress d by 6% in accuracy at the same compression ratio. SP 3 has also proven effective with other models, including OPT and Llama. Our data and code are available at ours repo.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites Reducing Transformer Depth on Demand with Structured Dropout
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites TinyBERT: Distilling BERT for Natural Language Understanding
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites Reducing Transformer Depth on Demand with Structured Dropout
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. SAM Decoding: Speculative Decoding via Suffix AutomatonYuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang et al.ACL 2025

Builds on31

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  3. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  4. LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao WangNeurIPS 2023 · 994 citations
  5. Reducing Transformer Depth on Demand with Structured DropoutAngela Fan, Edouard Grave, Armand JoulinICLR 2020 · 695 citations

Related papers

  1. Structured Pruning for Efficient Generative Pre-trained Language ModelsChaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei et al.ACL Findings 2023 · 13 citations
  2. Structured Pruning of Large Language ModelsZiheng Wang, Jeremy Wohlwend, Tao LeiEMNLP 2020 · 88 citations
  3. DISP-LLM: Dimension-Independent Structural Pruning for Large Language ModelsShangqian Gao, Chi-Heng Lin, Ting Hua, Zheng Tang et al.NeurIPS 2024 · 42 citations
  4. MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal ProjectionBokai Lin, Zihao Zeng, Zipeng Xiao, Siqi Kou et al.ICLR 2025
  5. Accurate Retraining-free Pruning for Pretrained Encoder-based Language ModelsSeungcheol Park, Hojun Choi, U KangICLR 2024 · 14 citations
Dusk over the sea between two cliffs drawn in fine vertical lines