EMNLP Findings2020
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior
Abstract
Traditional (unstructured) pruning methods for a Transformer model focus on regularizing the individual weights by penalizing them toward zero. In this work, we explore spectralnormalized identity priors (SNIP), a structured pruning approach that penalizes an entire residual module in a Transformer model toward an identity mapping. Our method identifies and discards unimportant non-linear mappings in the residual connections by applying a thresholding operator on the function norm. It is applicable to any structured module, including a single attention head, an entire attention block, or a feed-forward subnetwork. Furthermore, we introduce spectral normalization to stabilize the distribution of the post-activation values of the Transformer layers, further improving the pruning effectiveness of the proposed methodology. We conduct experiments with BERT on 5 GLUE benchmark tasks to demonstrate that SNIP achieves effective pruning results while maintaining comparable performance. Specifically, we improve the performance over the state-of-the-art by 0.5 to 1.0% on average at 50% compression ratio.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
13 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TinyBERT: Distilling BERT for Natural Language Understanding cites Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
- TinyBERT: Distilling BERT for Natural Language Understanding cites Reducing Transformer Depth on Demand with Structured Dropout
- TinyBERT: Distilling BERT for Natural Language Understanding cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices cites TinyBERT: Distilling BERT for Natural Language Understanding
- Learned Token Pruning for Transformers cites TinyBERT: Distilling BERT for Natural Language Understanding
- Learned Token Pruning for Transformers cites Movement Pruning: Adaptive Sparsity by Fine-Tuning
- Learned Token Pruning for Transformers cites Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
- Learned Token Pruning for Transformers cites Reducing Transformer Depth on Demand with Structured Dropout
- Learned Token Pruning for Transformers cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- Gradient-Free Structured Pruning with Unlabeled Data cites A Fast Post-Training Pruning Framework for Transformers
- Gradient-Free Structured Pruning with Unlabeled Data cites Movement Pruning: Adaptive Sparsity by Fine-Tuning
- Gradient-Free Structured Pruning with Unlabeled Data cites Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
- Gradient-Free Structured Pruning with Unlabeled Data cites Reducing Transformer Depth on Demand with Structured Dropout
- Gradient-Free Structured Pruning with Unlabeled Data cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- A Fast Post-Training Pruning Framework for Transformers cites Movement Pruning: Adaptive Sparsity by Fine-Tuning
- A Fast Post-Training Pruning Framework for Transformers cites Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
- A Fast Post-Training Pruning Framework for Transformers cites Reducing Transformer Depth on Demand with Structured Dropout
- A Fast Post-Training Pruning Framework for Transformers cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- Movement Pruning: Adaptive Sparsity by Fine-Tuning cites TinyBERT: Distilling BERT for Natural Language Understanding
- Movement Pruning: Adaptive Sparsity by Fine-Tuning cites Reducing Transformer Depth on Demand with Structured Dropout
- A Survey on Model Compression and Acceleration for Pretrained Language Models cites TinyBERT: Distilling BERT for Natural Language Understanding
- A Survey on Model Compression and Acceleration for Pretrained Language Models cites Learned Token Pruning for Transformers
- A Survey on Model Compression and Acceleration for Pretrained Language Models cites Movement Pruning: Adaptive Sparsity by Fine-Tuning
- A Survey on Model Compression and Acceleration for Pretrained Language Models cites Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
- A Survey on Model Compression and Acceleration for Pretrained Language Models cites Reducing Transformer Depth on Demand with Structured Dropout
- A Survey on Model Compression and Acceleration for Pretrained Language Models cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Cited by top-tier papers13
- A Fast Post-Training Pruning Framework for TransformersWoosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun et al.NeurIPS 2022 · 247 citations
- Learned Token Pruning for TransformersSehoon Kim, Sheng Shen, David Thorsley, Amir Gholami et al.KDD 2022 · 97 citations
- A Survey on Model Compression and Acceleration for Pretrained Language ModelsCanwen Xu, Julian J. McAuleyAAAI 2023 · 96 citations
- Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization SpaceArnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu et al.CVPR 2022 · 64 citations
- Gradient-Free Structured Pruning with Unlabeled DataAzade Nova, Hanjun Dai, Dale SchuurmansICML 2023 · 38 citations
Builds on7
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- Reducing Transformer Depth on Demand with Structured DropoutAngela Fan, Edouard Grave, Armand JoulinICLR 2020 · 695 citations
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu et al.ACL 2020 · 660 citations
- Movement Pruning: Adaptive Sparsity by Fine-TuningVictor Sanh, Thomas Wolf, Alexander M. RushNeurIPS 2020 · 656 citations
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTSheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma et al.AAAI 2020 · 656 citations
Related papers
- When BERT Plays the Lottery, All Tickets Are WinningSai Prasanna, Anna Rogers, Anna RumshiskyEMNLP 2020 · 114 citations
- Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer InferenceJunyan Li, Li Lyna Zhang, Jiahang Xu, Yujing Wang et al.KDD 2023 · 12 citations
- Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional ActivationsBowen Shen, Zheng Lin, Daren Zha, Wei Liu et al.ACL Findings 2024 · 3 citations
- Two-Stage Regularization-Based Structured Pruning for LLMsMingkuan Feng, Jinyang Wu, Siyuan Liu, Shuai Zhang et al.ACL 2026 · 3 citations
- Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured PruningBingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li et al.EMNLP Findings 2020 · 39 citations
