Lune

ACL Findings2023

SlowBERT: Slow-down Attacks on Input-adaptive Multi-exit BERT

Shengyao Zhang, Xudong Pan, Mi Zhang, Min Yang

Abstract

For pretrained language models such as Google's BERT, recent research designs several input-adaptive inference mechanisms to improve the efficiency on cloud and edge devices.In this paper, we reveal a new attack surface on input-adaptive multi-exit BERT, where the adversary imperceptibly modifies the input texts to drastically increase the average inference cost.Our proposed slow-down attack called SlowBERT integrates a new rank-andsubstitute adversarial text generation algorithm to efficiently search for the perturbation which maximally delays the exiting time.With no direct access to the model internals, we further devise a time-based approximation algorithm to infer the exit position as the loss oracle.Our extensive evaluation on two popular instances of multi-exit BERT for GLUE classification tasks validates the effectiveness of SlowBERT.In the worst case, SlowBERT increases the inference cost by 4.57×, which would strongly hurt the service quality of multi-exit BERT in practice, e.g., increasing the real-time cloud services' response time for online users.Embeddings Layer-1 Layer-2 Layer-3 Layer-K … IC-K IC-3 IC-2 IC-1 "a fast, funny, highly enjoyable movie."(b) Patience-based (C=2) Embeddings Layer-1 Layer-2 Layer-3 Layer-K … IC-K IC-3 IC-2 IC-1 "a fast, funny, highly enjoyable movie."(a) Confidence-based (T=0.8)patience count=2 POS: 0.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • NaturalSloth: Revisiting Denial-of-Service Attacks on Large Language Models cites VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service
  • BERT Loses Patience: Fast and Robust Inference with Early Exit cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • BERT Loses Patience: Fast and Robust Inference with Early Exit cites FastBERT: a Self-distilling BERT with Adaptive Inference Time
  • VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service cites BERT Loses Patience: Fast and Robust Inference with Early Exit
  • FastBERT: a Self-distilling BERT with Adaptive Inference Time cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment cites TextBugger: Generating Adversarial Text Against Real-world Applications
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a ServiceXiasi Wang, Tianliang Yao, Simin Chen, Runqi Wang et al.ACL 2025 · 3 citations
  2. NaturalSloth: Revisiting Denial-of-Service Attacks on Large Language ModelsYiming Chen, Zexin Li, Xianghu Yue, Robby T. Tan et al.ACL 2026

Builds on11

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  2. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and EntailmentDi Jin, Zhijing Jin, Joey Tianyi Zhou, Peter SzolovitsAAAI 2020 · 1,333 citations
  3. TextBugger: Generating Adversarial Text Against Real-world ApplicationsJinfeng Li, Shouling Ji, Tianyu Du, Bo Li et al.NDSS 2019 · 876 citations
  4. BERT Loses Patience: Fast and Robust Inference with Early ExitWangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley et al.NeurIPS 2020 · 473 citations
  5. FastBERT: a Self-distilling BERT with Adaptive Inference TimeWeijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao et al.ACL 2020 · 257 citations

Related papers

  1. Dynamic Transformers Provide a False Sense of EfficiencyYiming Chen, Simin Chen, Zexin Li, Wei Yang et al.ACL 2023 · 5 citations
  2. BERT Lost Patience Won't Be Robust to Adversarial SlowdownZachary Coalson, Gabriel Ritter, Rakesh Bobba, Sanghyun HongNeurIPS 2023 · 7 citations
  3. A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network InferenceSanghyun Hong, Yigitcan Kaya, Ionut-Vlad Modoranu, Tudor DumitrasICLR 2021 · 85 citations
  4. LeeBERT: Learned Early Exit for BERT with cross-level optimizationWei ZhuACL 2021
  5. BERT-ATTACK: Adversarial Attack Against BERT Using BERTLinyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue et al.EMNLP 2020 · 529 citations
Dusk over the sea between two cliffs drawn in fine vertical lines