Lune

EMNLP Findings2020

Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning

Bingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li, Zhengang Li, Hang Liu, Caiwen Ding

Abstract

Pre-trained large-scale language models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. However, the limited weight storage and computational speed on hardware platforms have impeded the popularity of pre-trained models, especially in the era of edge computing. In this work, we propose an efficient transformer-based large-scale language representation using hardware-friendly block structure pruning. We incorporate the reweighted group Lasso into block-structured pruning for optimization. Besides the significantly reduced weight storage and computation, the proposed approach achieves high compression rates. Experimental results on different models (BERT, RoBERTa, and DistilBERT) on the General Language Understanding Evaluation (GLUE) benchmark tasks show that we achieve up to 5.0× with zero or minor accuracy degradation on certain task(s). Our proposed method is also orthogonal to existing compact pre-trained language models such as DistilBERT using knowledge distillation, since a further 1.79× average compression rate can be achieved on top of DistilBERT with zero or minor accuracy degradation. It is suitable to deploy the final compressed model on resource-constrained edge devices. We share the related codes and models at: https://bi t.ly/3cvs2N2

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

10 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • Learned Token Pruning for Transformers cites TinyBERT: Distilling BERT for Natural Language Understanding
  • Learned Token Pruning for Transformers cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • Gradient-Free Structured Pruning with Unlabeled Data cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • Gradient-Free Structured Pruning with Unlabeled Data cites A Fast Post-Training Pruning Framework for Transformers
  • A Fast Post-Training Pruning Framework for Transformers cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers10

  1. A Fast Post-Training Pruning Framework for TransformersWoosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun et al.NeurIPS 2022 · 247 citations
  2. Sparse is Enough in Scaling TransformersSebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser et al.NeurIPS 2021 · 127 citations
  3. Learned Token Pruning for TransformersSehoon Kim, Sheng Shen, David Thorsley, Amir Gholami et al.KDD 2022 · 97 citations
  4. Green AI: Do Deep Learning Frameworks Have Different Costs?Stefanos Georgiou, Maria Kechagia, Tushar Sharma, Federica Sarro et al.ICSE 2022 · 90 citations
  5. Gradient-Free Structured Pruning with Unlabeled DataAzade Nova, Hanjun Dai, Dale SchuurmansICML 2023 · 38 citations

Builds on3

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  2. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  3. UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-TrainingHangbo Bao, Li Dong, Furu Wei, Wenhui Wang et al.ICML 2020 · 423 citations

Related papers

  1. ROSITA: Refined BERT cOmpreSsion with InTegrAted techniquesYuanxin Liu, Zheng Lin, Fengcheng YuanAAAI 2021 · 22 citations
  2. The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language ModelsEldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar et al.EMNLP 2022 · 4 citations
  3. DynaBERT: Dynamic BERT with Adaptive Width and DepthLu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang et al.NeurIPS 2020 · 401 citations
  4. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained TransformersWenhui Wang, Furu Wei, Li Dong, Hangbo Bao et al.NeurIPS 2020 · 2,727 citations
  5. EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge DistillationChenhe Dong, Guangrun Wang, Hang Xu, Jiefeng Peng et al.EMNLP Findings 2021 · 20 citations
Dusk over the sea between two cliffs drawn in fine vertical lines