ACL Findings2022
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, Furu Wei
摘要
As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e.g., search history, medical record, bank account). Privacy-preserving inference of transformer models is on the demand of cloud service users. To protect privacy, it is an attractive choice to compute only with ciphertext in homomorphic encryption (HE). However, enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks, which are not supported by current HE tools yet. In this work, we introduce THE-X, an approximation approach for transformers, which enables privacy-preserving inference of pre-trained models developed by popular frameworks. THE-X proposes a workflow to deal with complex computation in transformer networks, including all the non-polynomial functions like GELU, softmax, and Layer-Norm. Experiments reveal our proposed THE-X can enable transformer inference on encrypted data for different downstream tasks, all with negligible performance drop but enjoying the theory-guaranteed privacy-preserving advantage.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 37 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- SOFT: Softmax-free Transformer with Linear Complexity 引用了 Language Models are Few-Shot Learners
- Iron: Private Inference on Transformers 引用了 Deep Learning with Differential Privacy
- Split-and-Denoise: Protect large language model inference with local differential privacy 引用了 Iron: Private Inference on Transformers
- MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention 引用了 SOFT: Softmax-free Transformer with Linear Complexity
- MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention 引用了 Iron: Private Inference on Transformers
- TextHide: Tackling Data Privacy for Language Understanding Tasks 引用了 Deep Learning with Differential Privacy
- Nimbus: Secure and Efficient Two-Party Inference for Transformers 引用了 Iron: Private Inference on Transformers
- Nimbus: Secure and Efficient Two-Party Inference for Transformers 引用了 MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention
- Nimbus: Secure and Efficient Two-Party Inference for Transformers 引用了 BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
- BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers 引用了 Iron: Private Inference on Transformers
- Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness 引用了 Deep Learning with Differential Privacy
引用它的顶会 Paper37
- Iron: Private Inference on TransformersMeng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing 等NeurIPS 2022 · 209 次引用
- BOLT: Privacy-Preserving, Accurate and Efficient Inference for TransformersQi Pang, Jinhao Zhu, Helen Möllering, Wenting Zheng 等S&P 2024 · 149 次引用
- Split-and-Denoise: Protect large language model inference with local differential privacyPeihua Mai, Ran Yan, Zhe Huang, Youjia Yang 等ICML 2024 · 41 次引用
- MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous AttentionWenxuan Zeng, Meng Li, Wenjie Xiong, Tong Tong 等ICCV 2023 · 38 次引用
- Nimbus: Secure and Efficient Two-Party Inference for TransformersZhengyi Li, Kang Yang, Jin Tan, Wen-jie Lu 等NeurIPS 2024 · 34 次引用
它引用的 Paper5
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah 等NeurIPS 2020 · 64,255 次引用
- Deep Learning with Differential PrivacyMartín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan 等CCS 2016 · 7,620 次引用
- SOFT: Softmax-free Transformer with Linear ComplexityJiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu 等NeurIPS 2021 · 232 次引用
- Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and FairnessLingjuan Lyu, Xuanli He, Yitong LiEMNLP Findings 2020 · 51 次引用
- TextHide: Tackling Data Privacy for Language Understanding TasksYangsibo Huang, Zhao Song, Danqi Chen, Kai Li 等EMNLP Findings 2020 · 49 次引用
相关 Paper
- Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic EncryptionItamar Zimerman, Moran Baruch, Nir Drucker, Gilad Ezov 等ICML 2024 · 26 次引用
- THOR: Secure Transformer Inference with Homomorphic EncryptionJungho Moon, Dongwoo Yoo, Xiaoqian Jiang, Miran KimCCS 2025 · 1 次引用
- Encryption-Friendly LLM ArchitectureDonghwan Rho, Taeseong Kim, Minje Park, Jung Woo Kim 等ICLR 2025
- Primer: Fast Private Transformer Inference on Encrypted DataMengxin Zheng, Qian Lou, Lei JiangDAC 2023 · 26 次引用
- Powerformer: Efficient and High-Accuracy Privacy-Preserving Language Model with Homomorphic EncryptionDongjin Park, Eunsang Lee, Joon-Woo LeeACL 2025 · 13 次引用
