ACL Findings2023
SlowBERT: Slow-down Attacks on Input-adaptive Multi-exit BERT
Shengyao Zhang, Xudong Pan, Mi Zhang, Min Yang
Abstract
For pretrained language models such as Google's BERT, recent research designs several input-adaptive inference mechanisms to improve the efficiency on cloud and edge devices.In this paper, we reveal a new attack surface on input-adaptive multi-exit BERT, where the adversary imperceptibly modifies the input texts to drastically increase the average inference cost.Our proposed slow-down attack called SlowBERT integrates a new rank-andsubstitute adversarial text generation algorithm to efficiently search for the perturbation which maximally delays the exiting time.With no direct access to the model internals, we further devise a time-based approximation algorithm to infer the exit position as the loss oracle.Our extensive evaluation on two popular instances of multi-exit BERT for GLUE classification tasks validates the effectiveness of SlowBERT.In the worst case, SlowBERT increases the inference cost by 4.57×, which would strongly hurt the service quality of multi-exit BERT in practice, e.g., increasing the real-time cloud services' response time for online users.Embeddings Layer-1 Layer-2 Layer-3 Layer-K … IC-K IC-3 IC-2 IC-1 "a fast, funny, highly enjoyable movie."(b) Patience-based (C=2) Embeddings Layer-1 Layer-2 Layer-3 Layer-K … IC-K IC-3 IC-2 IC-1 "a fast, funny, highly enjoyable movie."(a) Confidence-based (T=0.8)patience count=2 POS: 0.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- NaturalSloth: Revisiting Denial-of-Service Attacks on Large Language Models cites VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service
- BERT Loses Patience: Fast and Robust Inference with Early Exit cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- BERT Loses Patience: Fast and Robust Inference with Early Exit cites FastBERT: a Self-distilling BERT with Adaptive Inference Time
- VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service cites BERT Loses Patience: Fast and Robust Inference with Early Exit
- FastBERT: a Self-distilling BERT with Adaptive Inference Time cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment cites TextBugger: Generating Adversarial Text Against Real-world Applications
Cited by top-tier papers2
- VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a ServiceXiasi Wang, Tianliang Yao, Simin Chen, Runqi Wang et al.ACL 2025 · 3 citations
- NaturalSloth: Revisiting Denial-of-Service Attacks on Large Language ModelsYiming Chen, Zexin Li, Xianghu Yue, Robby T. Tan et al.ACL 2026
Builds on11
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and EntailmentDi Jin, Zhijing Jin, Joey Tianyi Zhou, Peter SzolovitsAAAI 2020 · 1,333 citations
- TextBugger: Generating Adversarial Text Against Real-world ApplicationsJinfeng Li, Shouling Ji, Tianyu Du, Bo Li et al.NDSS 2019 · 876 citations
- BERT Loses Patience: Fast and Robust Inference with Early ExitWangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley et al.NeurIPS 2020 · 473 citations
- FastBERT: a Self-distilling BERT with Adaptive Inference TimeWeijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao et al.ACL 2020 · 257 citations
Related papers
- Dynamic Transformers Provide a False Sense of EfficiencyYiming Chen, Simin Chen, Zexin Li, Wei Yang et al.ACL 2023 · 5 citations
- BERT Lost Patience Won't Be Robust to Adversarial SlowdownZachary Coalson, Gabriel Ritter, Rakesh Bobba, Sanghyun HongNeurIPS 2023 · 7 citations
- A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network InferenceSanghyun Hong, Yigitcan Kaya, Ionut-Vlad Modoranu, Tudor DumitrasICLR 2021 · 85 citations
- LeeBERT: Learned Early Exit for BERT with cross-level optimizationWei ZhuACL 2021
- BERT-ATTACK: Adversarial Attack Against BERT Using BERTLinyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue et al.EMNLP 2020 · 529 citations
