SlowBERT: Slow-down Attacks on Input-adaptive Multi-exit BERT

For pretrained language models such as Google's BERT, recent research designs several input-adaptive inference mechanisms to improve the efficiency on cloud and edge devices. In this paper, we reveal a new attack surface on input-adaptive multi-exit BERT, where the adversary imperceptibly modifies the input texts to drastically increase the average inference cost. Our proposed slow-down attack called SlowBERT integrates a new rank-andsubstitute adversarial text generation algorithm to efficiently search for the perturbation which maximally delays the exiting time. With no direct access to the model internals, we further devise a time-based approximation algorithm to infer the exit position as the loss oracle. Our extensive evaluation on two popular instances of multi-exit BERT for GLUE classification tasks validates the effectiveness of SlowBERT. In the worst case, SlowBERT increases the inference cost by 4.57×, which would strongly hurt the service quality of multi-exit BERT in practice, e.g., increasing the real-time cloud services' response time for online users. Embeddings Layer-1 Layer-2 Layer-3 Layer-K … IC-K IC-3 IC-2 IC-1 "a fast, funny, highly enjoyable movie." (b) Patience-based (C=2) Embeddings Layer-1 Layer-2 Layer-3 Layer-K … IC-K IC-3 IC-2 IC-1 "a fast, funny, highly enjoyable movie." (a) Confidence-based (T=0.8