Lune

ACL Findings2021

EBERT: Efficient BERT Inference with Dynamic Structured Pruning

Zejian Liu, Fanrong Li, Gang Li, Jian Cheng

Abstract

Pruning has been demonstrated as an effective way of reducing computational complexity for deep networks, especially CNNs for computer vision tasks. In this paper, we investigate the opportunity to accelerate the inference of large-scale pre-trained language model via pruning. We propose EBERT, a dynamic structured pruning algorithm for efficient BERT inference. Unlike previous methods that randomly prune the model weights for static inference, EBERT dynamically determines and prunes the unimportant heads in multi-head self-attention layers and the unimportant structured computations in feed-forward network for each input sample at run-time. Experimental results show that our proposed EBERT outperforms other state-of-the-art methods on different tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Learned Token Pruning for Transformers cites TinyBERT: Distilling BERT for Natural Language Understanding
  • Learned Token Pruning for Transformers cites Movement Pruning: Adaptive Sparsity by Fine-Tuning
  • Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models cites TinyBERT: Distilling BERT for Natural Language Understanding
  • Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models cites A Fast Post-Training Pruning Framework for Transformers
  • Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models cites Movement Pruning: Adaptive Sparsity by Fine-Tuning
  • Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models cites Language Models are Few-Shot Learners
  • Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models cites LLM-Pruner: On the Structural Pruning of Large Language Models
  • BERT Loses Patience: Fast and Robust Inference with Early Exit cites Language Models are Few-Shot Learners
  • The Need for Speed: Pruning Transformers with One Recipe cites A Fast Post-Training Pruning Framework for Transformers
  • The Need for Speed: Pruning Transformers with One Recipe cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • The Need for Speed: Pruning Transformers with One Recipe cites Movement Pruning: Adaptive Sparsity by Fine-Tuning
  • The Need for Speed: Pruning Transformers with One Recipe cites LLM-Pruner: On the Structural Pruning of Large Language Models
  • A Fast Post-Training Pruning Framework for Transformers cites BERT Loses Patience: Fast and Robust Inference with Early Exit
  • A Fast Post-Training Pruning Framework for Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • A Fast Post-Training Pruning Framework for Transformers cites Movement Pruning: Adaptive Sparsity by Fine-Tuning
  • A Fast Post-Training Pruning Framework for Transformers cites Language Models are Few-Shot Learners
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
  • Movement Pruning: Adaptive Sparsity by Fine-Tuning cites TinyBERT: Distilling BERT for Natural Language Understanding
  • Movement Pruning: Adaptive Sparsity by Fine-Tuning cites Language Models are Few-Shot Learners
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites TinyBERT: Distilling BERT for Natural Language Understanding
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites A Fast Post-Training Pruning Framework for Transformers
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao WangNeurIPS 2023 · 994 citations
  2. A Fast Post-Training Pruning Framework for TransformersWoosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun et al.NeurIPS 2022 · 247 citations
  3. Learned Token Pruning for TransformersSehoon Kim, Sheng Shen, David Thorsley, Amir Gholami et al.KDD 2022 · 97 citations
  4. The Need for Speed: Pruning Transformers with One RecipeSamir Khaki, Konstantinos N. PlataniotisICLR 2024 · 17 citations
  5. Accurate Retraining-free Pruning for Pretrained Encoder-based Language ModelsSeungcheol Park, Hojun Choi, U KangICLR 2024 · 14 citations

Builds on9

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  4. Movement Pruning: Adaptive Sparsity by Fine-TuningVictor Sanh, Thomas Wolf, Alexander M. RushNeurIPS 2020 · 656 citations
  5. BERT Loses Patience: Fast and Robust Inference with Early ExitWangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley et al.NeurIPS 2020 · 473 citations

Related papers

  1. EarlyBERT: Efficient BERT Training via Early-bird Lottery TicketsXiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan et al.ACL 2021
  2. DynaBERT: Dynamic BERT with Adaptive Width and DepthLu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang et al.NeurIPS 2020 · 401 citations
  3. COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language ModelsBowen Shen, Zheng Lin, Yuanxin Liu, Zhengxiao Liu et al.EMNLP 2022 · 3 citations
  4. FastBERT: a Self-distilling BERT with Adaptive Inference TimeWeijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao et al.ACL 2020 · 257 citations
  5. PDP: Parameter-free Differentiable Pruning is All You NeedMinsik Cho, Saurabh Adya, Devang NaikNeurIPS 2023 · 24 citations
Dusk over the sea between two cliffs drawn in fine vertical lines