Lune

ACL Findings2023

Operator Selection and Ordering in a Pipeline Approach to Efficiency Optimizations for Transformers

Ji Xin, Raphael Tang, Zhiying Jiang, Yaoliang Yu, Jimmy Lin

Abstract

There exists a wide variety of efficiency methods for natural language processing (NLP) tasks, such as pruning, distillation, dynamic inference, quantization, etc. From a different perspective, we can consider an efficiency method as an operator applied on a model. Naturally, we may construct a pipeline of operators, i.e., to apply multiple efficiency methods on the model sequentially. In this paper, we study the plausibility of this idea, and more importantly, the commutativity and cumulativeness of efficiency operators. We make two interesting observations from our experiments: (1) The operators are commutative-the order of efficiency methods within the pipeline has little impact on the final results; (2) The operators are also cumulative-the final results of combining several efficiency methods can be estimated by combining the results of individual methods. These observations deepen our understanding of efficiency operators and provide useful guidelines for building them in real-world applications.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
  • TinyBERT: Distilling BERT for Natural Language Understanding cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
  • MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices cites TinyBERT: Distilling BERT for Natural Language Understanding
  • PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination cites Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on10

  1. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  2. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu et al.ACL 2020 · 660 citations
  3. Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTSheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma et al.AAAI 2020 · 656 citations
  4. BERT Loses Patience: Fast and Robust Inference with Early ExitWangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley et al.NeurIPS 2020 · 473 citations
  5. PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector EliminationSaurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan T. Chakaravarthy et al.ICML 2020 · 260 citations

Related papers

  1. Reinforced Multi-Teacher Selection for Knowledge DistillationFei Yuan, Linjun Shou, Jian Pei, Wutao Lin et al.AAAI 2021 · 155 citations
  2. Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model CompressionMinjun Kim, Jaehyeon Choi, Hyunwoo Yang, Jongjin Kim et al.ICLR 2026 · 5 citations
  3. Length-Adaptive Distillation: Customizing Small Language Model for Dynamic Token PruningChang Liu, Chongyang Tao, Jianxin Liang, Jiazhan Feng et al.EMNLP Findings 2023
  4. NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language ModelsAmit Dhurandhar, Tejaswini Pedapati, Ronny Luss, Soham Dan et al.ACL Findings 2024 · 3 citations
  5. Bag of Tricks for Optimizing Transformer EfficiencyYe Lin, Yanyang Li, Tong Xiao, Jingbo ZhuEMNLP Findings 2021 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines